Fusion of sensor data for detecting interactions at an inventory location

ABSTRACT

Items may be stowed in an inventory location, such as a shelf. An interaction may take place where one of the items is picked from or placed on the inventory location. Sensor data can be acquired from two or more sensors, such as cameras or weight sensors, configured to collect such sensor data for the inventory location. Hypotheses to describe interactions at the inventory location may be determined using sensor data from each different sensor. However, the confidence value for such hypotheses may not be reliable. Fusion of sensor data may be performed where sets of hypotheses from the different sensors are combined and the resulting hypotheses are more accurate.

PRIORITY

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 14/675,167, filed on Mar. 31, 2015 entitled“Multiple Sensor Data Fusion System”, which is hereby incorporated byreference in its entirety.

BACKGROUND

Retailers, wholesalers, and other product distributors typicallymaintain an inventory of various items that may be ordered, purchased,leased, borrowed, rented, viewed, and so forth, by clients or customers.For example, an e-commerce website may maintain inventory in afulfillment center. When a customer orders an item, the item is pickedfrom inventory, routed to a packing station, packed, and shipped to thecustomer. Likewise, physical stores maintain inventory in customeraccessible areas, such as in a shopping area, and customers can pickitems from inventory and take them to a cashier for purchase, rental,and so forth.

Many physical stores also maintain inventory in a storage area,fulfillment center, or other facility that can be used to replenishinventory located in the shopping areas or to satisfy orders for itemsthat are placed through other channels (e.g., e-commerce). Otherexamples of entities that maintain facilities holding inventory includelibraries, museums, rental centers, and so forth. In each instance, foran item to be moved from one location to another, it is picked from itscurrent location and transitioned to a new location. It is oftendesirable to monitor quantity of inventory within the facility.

BRIEF DESCRIPTION OF FIGURES

The detailed description is set forth with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical items or features.

FIG. 1 is a block diagram illustrating a material handling facility(facility) having various sensors, according to some implementations.

FIG. 2 is a block diagram illustrating additional details of thefacility, according to some implementations.

FIG. 3 is a block diagram of a server configured to support operation ofthe facility, according to some implementations.

FIG. 4 is a block diagram of additional data that may be used by theserver to support operation of the facility, according to someimplementations.

FIG. 5 illustrates a side view of an inventory location that includesvarious sensors, according to some implementations.

FIG. 6 illustrates enlarged top and side views of a portion of theinventory location, according to some implementations.

FIG. 7 illustrates an overhead view of partitioned areas at theinventory location, the shadows of objects on the partitioned area, anda weight distribution of the inventory location, according to someimplementations.

FIG. 8 illustrates a front view of the inventory location before andafter removal of an item from the inventory location, according to someimplementations.

FIG. 9 depicts a flow diagram of a process for determining interactiondata using different sets of hypotheses, according to someimplementations.

FIG. 10 depicts a flow diagram of a process for generating interactiondata indicative of an interaction such as a pick or place of an item,according to some implementations.

FIG. 11 depicts a flow diagram of another process for generatinginteraction data, according to some implementations.

FIG. 12 depicts a flow diagram of a process of generating weight data,according to some implementations.

While implementations are described herein by way of example, thoseskilled in the art will recognize that the implementations are notlimited to the examples or figures described. It should be understoodthat the figures and detailed description thereto are not intended tolimit implementations to the particular form disclosed but, on thecontrary, the intention is to cover all modifications, equivalents, andalternatives falling within the spirit and scope as defined by theappended claims. The headings used herein are for organizationalpurposes only and are not meant to be used to limit the scope of thedescription or the claims. As used throughout this application, the word“may” is used in a permissive sense (i.e., meaning having the potentialto), rather than the mandatory sense (i.e., meaning must). Similarly,the words “include,” “including,” and “includes” mean including, but notlimited to.

DETAILED DESCRIPTION

At a material handling facility (facility) items may be stowed or heldat inventory locations. A single inventory location, such as a shelf orrack, may hold several different types of items. During use of thefacility, interactions such as a pick (removal) or place (stowage) of anitem from the inventory locations may take place.

The facility may include, or have access to, an inventory managementsystem. The inventory management system may be configured to maintaininformation about items, users, condition of the facility, and so forth.For example, the inventory management system may maintain dataindicative of a number of items at a particular inventory location, whatitems a particular user is ordered to pick, how many items have beenpicked or placed at the inventory location, requests for assistance,environmental status of the facility, and so forth.

During operation, the inventory management system may utilize sensordata acquired from a plurality of sensors at the facility. For example,weight sensors at the inventory location may gather weight data, cameraswith fields of view that include the inventory location may gather imagedata, and so forth.

This disclosure describes systems and techniques for processing sensordata from multiple sensors to generate interaction data indicative ofinteractions at the facility. Data fusion techniques are used to combinehypotheses data produced using sensor data acquired from one or moresensors.

The hypotheses data may be generated which describes variousinteractions that could occur. The hypotheses data may be based on oneor more of item data, physical configuration data, sensor data,predicted data, and so forth. For example, the hypotheses data mayindicate a predicted change in quantity to one of more of the items at apredicted partitioned area of the inventory location. Each hypothesismay include a predicted value such as predicted weight distribution,predicted quantity, and so forth.

The item data may provide information about one or more of the items.This information may include the partitioned area(s) in which the itemis designated to be stowed, weight of an individual item, currentquantity on hand, and so forth.

A set of hypotheses based on the sensor data from a particular sensormay be determined. For example, a first set of hypotheses may begenerated that are consistent with the image data acquired by theimaging sensors. Likewise, a second set of hypotheses may be generatedfrom weight data acquired by the weight sensors.

Hypotheses in the hypotheses data that are inconsistent with the sensordata may be discarded or disregarded from the set of hypotheses. Forexample, image data that depicts a pick of an item may result in discardof hypotheses that describe the opposite situation of placement of anitem. Each hypothesis in a set may include a probability value. Theprobability may indicate how likely that particular hypothesis is to betrue. In some implementations, the probability may be implicit. Forexample, a hypothesis with no stated probability may be deemed to have aprobability of 1.0 of being true.

Confidence values may be determined for at least a portion of hypothesesin a set of hypotheses. The confidence value provides indicia of howlikely a particular hypothesis is to be true, with respect to anotherhypothesis in the same set. In one implementation, the hypotheses in aset may be sorted or ranked according to their respective probabilities.A first hypothesis having a first probability and a highest rank may bedetermined. Similarly, a second hypothesis having a second probabilityless than the first probability and second highest rank may bedetermined. The confidence level may comprise a difference or ratiobetween the first probability and the second probability. When theconfidence level associated with a particular hypothesis exceeds aconfidence threshold value, that hypothesis may be deemed the solution.The resulting solution may then be used to generate the interactiondata.

In some situations, a single set of hypotheses based on sensor data froma single sensor or group of sensors of the same type may not producehypotheses with a high enough confidence value. To generate a solutionwith a sufficiently high confidence level, two or more sets ofhypotheses may be combined. These sets of hypotheses may be generatedusing one or more of sensor data from different types of sensors, orprocessing the sensor data using different techniques.

Continuing the example above, the first set of hypotheses based on imagedata and the second set of hypotheses based on weight data may becombined to determine a third set of hypotheses. In someimplementations, the combination of sets of hypotheses may use Bayes'rule. For example, Bayes' rule may be used to aggregate the first set ofhypotheses (based on image data) and the second set of hypotheses (basedon weight data) by computing a joint probability of the hypothesesrepresentative of concurrent events.

The confidence values may be determined for the hypotheses that appearin the combined sets of hypotheses. For example, the confidence valuesmay be determined for the third set of hypotheses. The confidence valuemay be compared with the confidence threshold value to determine if theassociated hypothesis is deemed to have a “high confidence” or a “lowconfidence”. A “high confidence” hypothesis may be deemed suitable foruse as a solution, while a “low confidence” may call for additionalprocessing to produce a “high confidence” hypothesis. When theconfidence level associated with the particular hypothesis does notexceed the threshold value, additional sets of hypotheses may be used todetermine the solution. For example, when the third set of hypothesesdoes not include a hypothesis with a confidence value above thethreshold value, a fourth set of hypotheses may be generated using theimage data. In one implementation, the image data may be processed todetermine a count of items that were picked or placed from the inventorylocation. The fourth set of hypotheses may comprise the output of thisprocessing, indicating hypotheses that describe item quantities based onthe image data and the probability that a particular hypothesis is true.

The fourth set of hypotheses also based on the image data may becombined with the third set of hypotheses to generate a fifth set ofhypotheses. The confidence levels may be determined for at least aportion of the hypotheses in the fifth set. As above, when the firstranked hypothesis of the fifth set has a confidence value that is abovea threshold value, that first ranked hypothesis may be designated as thesolution. This solution may then be used to generate interaction data.For example, the interaction data may indicate the inventory location,item identifier indicative of a particular item, change in quantity ofthe item, and so forth.

The inventory management system may use the interaction data to maintainitem data about the items in the facility. For example, whereinteraction data indicates a pick of a particular quantity of aparticular item from a particular inventory location, the item dataindicative of quantity on hand of that particular item at thatparticular location may be decreased accordingly.

By using the devices and techniques described herein, operation of thefacility may be improved. Details about interactions between users anditems in the facility may be quickly and accurately determined. Forexample, as items are picked, placed, and so forth, information such asquantity on hand at a given time may be readily determined. As a result,the inventory management system may be able to quickly track what item auser has interacted with, maintain up-to-date item data, and so forth.

Illustrative System

An implementation of a material handling system 100 configured to storeand manage inventory items is illustrated in FIG. 1. A material handlingfacility 102 (facility) comprises one or more physical structures orareas within which one or more items 104(1), 104(2), . . . , 104(Q) maybe held. As used in this disclosure, letters in parenthesis such as“(Q)” indicate an integer value greater than or equal to zero. The items104 comprise physical goods, such as books, pharmaceuticals, repairparts, electronic gear, and so forth.

The facility 102 may include one or more areas designated for differentfunctions with regard to inventory handling. In this illustration, thefacility 102 includes a receiving area 106, a storage area 108, and atransition area 110.

The receiving area 106 may be configured to accept items 104, such asfrom suppliers, for intake into the facility 102. For example, thereceiving area 106 may include a loading dock at which trucks or otherfreight conveyances unload the items 104.

The storage area 108 is configured to store the items 104. The storagearea 108 may be arranged in various physical configurations. In oneimplementation, the storage area 108 may include one or more aisles 112.The aisle 112 may be configured with, or defined by, inventory locations114 on one or both sides of the aisle 112. The inventory locations 114may include one or more of shelves, racks, cases, cabinets, bins, floorlocations, or other suitable storage mechanisms for holding, supporting,or storing the items 104. The inventory locations 114 may be affixed tothe floor or another portion of the facility's 102 structure. Theinventory locations 114 may also be movable such that the arrangementsof aisles 112 may be reconfigurable. In some implementations, theinventory locations 114 may be configured to move independently of anoutside operator. For example, the inventory locations 114 may comprisea rack with a power source and a motor, operable by a computing deviceto allow the rack to move from one location within the facility 102 toanother.

One or more users 116(1), 116(2), . . . , 116(U) and totes 118(1),118(2), . . . , 118(T) or other material handling apparatuses may movewithin the facility 102. For example, the user 116 may move about withinthe facility 102 to pick or place the items 104 at various inventorylocations 114. For ease of transport, the items 104 may be carried bythe tote 118. The tote 118 is configured to carry or otherwise transportone or more items 104. For example, the tote 118 may include a basket,cart, bag, bin, and so forth. In other implementations, other materialhandling apparatuses such as robots, forklifts, cranes, aerial drones,and so forth, may move about the facility 102 picking, placing, orotherwise moving the items 104. For example, a robot may pick an item104 from a first inventory location 114(1) and move the item 104 to asecond inventory location 114(2). In some implementations, at least aportion of the tote 118 may be designated as an inventory location 114and may be equipped as described herein with weight sensors.

One or more sensors 120 may be configured to acquire information aboutevents at the facility 102. The sensors 120 may include, but are notlimited to, cameras, three-dimensional (3D) sensors, weight sensors,optical sensor arrays, proximity sensors, and so forth. The sensors 120may be stationary or mobile, relative to the facility 102. For example,the inventory locations 114 may contain weight sensors 120(6) to acquireweight data of items 104 stowed therein, cameras to acquire images ofpicking or placement of items 104 on shelves, optical sensor arrays todetect shadows of the user's 116 hands at the inventory locations 114,and so forth. In another example, the facility 102 may include camerasto obtain images of the user 116 or other objects in the facility 102.The sensors 120 are discussed in more detail below with regard to FIG.2.

While the storage area 108 is depicted as having one or more aisles 112,inventory locations 114 storing the items 104, sensors 120, and soforth, it is understood that the receiving area 106, the transition area110, or other areas of the facility 102 may be similarly equipped.Furthermore, the arrangement of the various areas within the facility102 is depicted functionally rather than schematically. For example, insome implementations, multiple different receiving areas 106, storageareas 108, and transition areas 110 may be interspersed rather thansegregated in the facility 102.

The facility 102 may include, or be coupled to, an inventory managementsystem 122. The inventory management system 122 is configured tointeract with users 116 or devices such as sensors 120, robots, materialhandling equipment, computing devices, and so forth, in one or more ofthe receiving area 106, the storage area 108, or the transition area110.

During operation of the facility 102, the sensors 120 may be configuredto provide sensor data 124, or information based on the sensor data 124,to the inventory management system 122. The sensor data 124 may includeweight data 126 obtained from weight sensors 120(6) or non-weight data128 obtained from other sensors 120 such as cameras 120(1), 3D sensors120(2), optical sensor arrays 120(13), proximity sensors 120(14), lightcurtains, and so forth. The sensors 120 are described in more detailbelow.

The weight data 126 comprises data generated by one or more weightsensors 120(6) configured to measure the weight of an inventory location114 that may stow the items 104. For example, the weight sensor 120(6)may comprise a load cell beneath a load that may include a shelf orplatform of the inventory location 114. By reading one or morecharacteristics of the load cell, the weight of the load may bedetermined.

The non-weight data 128 may comprise data generated by the non-weightsensors 120, such as cameras 120(1), 3D sensors 120(2), buttons 120(3),touch sensors 120(4), microphones 120(5), optical sensors 120(7), RFIDreaders 120(8), RF receivers 120(9), accelerometers 120(10), gyroscopes120(11), magnetometers 120(12), optical sensor arrays 120(13), proximitysensors 120(14), and so forth. For example, cameras 120(1) may bearranged to have a field of view (FOV) 130 that includes at least aportion of the inventory location 114. Continuing the example, thecamera 120(1) may be mounted above the inventory location 114 with theFOV 130 oriented to where the items 104 may be stowed during use.

The inventory management system 122 or other systems may use the sensordata 124 to track the location of objects within the facility 102,movement of the objects, or provide other functionality. Objects mayinclude, but are not limited to, items 104, users 116, totes 118, and soforth. For example, a series of images acquired by the camera 120(1) mayindicate removal by the user 116 of an item 104 from a particularlocation on the inventory location 114 and placement of the item 104 onor at least partially within the tote 118.

The facility 102 may be configured to receive different kinds of items104 from various suppliers and to store them until a customer orders orretrieves one or more of the items 104. A general flow of items 104through the facility 102 is indicated by the arrows of FIG. 1.Specifically, as illustrated in this example, items 104 may be receivedfrom one or more suppliers, such as manufacturers, distributors,wholesalers, and so forth, at the receiving area 106. In variousimplementations, the items 104 may include merchandise, commodities,perishables, or any suitable type of item 104, depending on the natureof the enterprise that operates the facility 102.

Upon being received from a supplier at the receiving area 106, the items104 may be prepared for storage in the storage area 108. For example, insome implementations, items 104 may be unpacked or otherwise rearranged.The inventory management system 122 may include one or more softwareapplications executing on a computer system to provide inventorymanagement functions. These inventory management functions may includemaintaining information indicative of the type, quantity, condition,cost, location, weight, or any other suitable parameters with respect tothe items 104. The items 104 may be stocked, managed, or dispensed interms of countable units, individual units, or multiple units, such aspackages, cartons, crates, pallets, or other suitable aggregations.Alternatively, some items 104, such as bulk products, commodities, andso forth, may be stored in continuous or arbitrarily divisible amountsthat may not be inherently organized into countable units. Such items104 may be managed in terms of measurable quantity such as units oflength, area, volume, weight, time, duration, or other dimensionalproperties characterized by units of measurement. Generally speaking, aquantity of an item 104 may refer to either a countable number ofindividual or aggregate units of an item 104 or a measurable amount ofan item 104, as appropriate.

After arriving through the receiving area 106, items 104 may be storedwithin the storage area 108. In some implementations, like items 104 maybe stored or displayed together in the inventory locations 114 such asin bins, on shelves, hanging from pegboards, and so forth. In thisimplementation, all items 104 of a given kind are stored in oneinventory location 114. In other implementations, like items 104 may bestored in different inventory locations 114. For example, to optimizeretrieval of certain items 104 having frequent turnover within a largephysical facility 102, those items 104 may be stored in severaldifferent inventory locations 114 to reduce congestion that might occurat a single inventory location 114.

When a customer order specifying one or more items 104 is received, oras a user 116 progresses through the facility 102, the correspondingitems 104 may be selected or “picked” from the inventory locations 114containing those items 104. In various implementations, item picking mayrange from manual to completely automated picking. For example, in oneimplementation, a user 116 may have a list of items 104 they desire andmay progress through the facility 102 picking items 104 from inventorylocations 114 within the storage area 108, and placing those items 104into a tote 118. In other implementations, employees of the facility 102may pick items 104 using written or electronic pick lists derived fromcustomer orders. These picked items 104 may be placed into the tote 118as the employee progresses through the facility 102.

After items 104 have been picked, they may be processed at a transitionarea 110. The transition area 110 may be any designated area within thefacility 102 where items 104 are transitioned from one location toanother or from one entity to another. For example, the transition area110 may be a packing station within the facility 102. When the item 104arrives at the transition area 110, the item 104 may be transitionedfrom the storage area 108 to the packing station. Information about thetransition may be maintained by the inventory management system 122.

In another example, if the items 104 are departing the facility 102, alist of the items 104 may be obtained and used by the inventorymanagement system 122 to transition responsibility for, or custody of,the items 104 from the facility 102 to another entity. For example, acarrier may accept the items 104 for transport with that carrieraccepting responsibility for the items 104 indicated in the list. Inanother example, a user 116 may purchase or rent the items 104 andremove the items 104 from the facility 102. During use of the facility102, the user 116 may move about the facility 102 to perform varioustasks, such as picking or placing the items 104 in the inventorylocations 114.

Objects such as an item 104, hand, robotic manipulator, retrieval tool,and so forth, may exhibit a shadow 132 with respect to the opticalsensor array 120(13) at an inventory location 114. The shadow 132 isillustrated with a dotted line in this figure. In one implementation,the optical sensor array 120(13) may be located below the item 104, suchas within a shelf upon or above which the item 104 is supported. Theshadow 132 may be cast upon the optical sensor array 120(13). Forexample, where the optical sensor array 120(13) is on a vertical wallbehind the items 104, the shadow 132 may comprise the shadow 132 cast onthat wall.

The optical sensor array 120(13) may comprise one or more sensors 120,such as optical sensors 120(7). The optical sensors 120(7) may bearranged in a regular, repeating, or periodic two-dimensionalarrangement such as a grid. Each of the optical sensors 120(7) may beconfigured to provide output indicative of a light intensity value. Forexample, the optical sensors 120(7) may generate an 8-bit valueindicative of an intensity of light ranging from value 255 indicatingmaximum intensity to value 0 indicating minimum intensity. In anotherimplementation, the light intensity value may be a 1-bit value of 0 or1.

A single optical sensor array 120(13) may be associated with severaldifferent items 104. For example, the inventory location 114 maycomprise a shelf that includes an optical sensor array 120(13). Theshelf may have sufficient space to allow for storage of severaldifferent kinds of items 104. Items 104 may be grouped together andplaced within a partitioned area 134. The partitioned area 134 maycomprise a lane or row of identical items 104 positioned one in front ofanother. For example, a left half of the shelf may store a first kind ofitem 104(1), while a right half of the shelf may store a second kind ofitem 104(2). The inventory management system 122 may be configured toaccess partition data indicative of which portion of the optical sensorarray 120(13), or an output thereof, is associated with a particularitem 104.

The optical sensor array 120(13) may generate non-weight data 128 suchas image data. The image data may comprise a plurality of pixels. Eachpixel may correspond to a position within the two-dimensionalarrangement of the optical sensors 120(7) and comprises the lightintensity value from the optical sensor 120(7) at the position. In someimplementations, the image data may comprise data from a subset of theoptical sensors 120(7) within the optical sensor array 120(13). Forexample, the image data may comprise information from the opticalsensors 120(7) corresponding to a particular partitioned area 134. Inanother example, image data from an optical sensor array 120(13) havinga plurality of partitioned areas 134 may be segmented into therespective partitioned areas 134 for further processing.

The inventory management system 122 is configured to use the sensor data124 and item data 136 to generate interaction data. The item data 136may include information about the item 104, such as weight, appearance,where the item 104 is stowed, and so forth. The interaction data mayprovide information about an interaction, such as a pick of an item 104from the inventory location 114, a place of an item 104 to the inventorylocation 114, a touch made to an item 104 at the inventory location 114,a gesture associated with an item 104 at the inventory location 114, andso forth. For example, the gesture may include the user 116 reachingtowards the item 104 held by the inventory location 114.

The interaction data may include one or more of the type of interaction,partitioned area 134 involved, item identifier, quantity change to theitem 104, user identifier, and so forth. The interaction data may thenbe used to further update the item data 136. For example, the quantityof items 104 on hand at a particular partitioned area 134 may be changedbased on an interaction that picks or places one or more items 104.

The inventory management system 122 may use the sensor data 124 todetermine the interaction. Weight characteristics about an interactionmay be determined using the weight data 126. These weightcharacteristics may include weight before an interaction, weight afteran interaction, amount of change in the weight of the inventory location114, weight distribution of the inventory location 114 at the weightsensors 120(6), a change in the weight distribution of the inventorylocation 114, and so forth. For example, an inventory location 114 maystow a single type of item 104. A count of the quantity of items 104picked or placed may be determined by dividing the change in weightassociated with an interaction by the weight of a single item 104 asstored in the item data 136.

In some implementations, a single inventory location 114 may stowseveral different types of items 104, such as arranged in differentpartitioned areas 134 as described above. The inventory managementsystem 122 may use the weight data 126 to determine weightcharacteristics, and use those weight characteristics to identify theitem 104 that was picked or placed. For example, a change in the weightdistribution, direction and distance of a change in the center-of-massweight, and so forth, may be indicative of a pick or place of an item104 from a particular partitioned area 134. The inventory managementsystem 122 may also use the weight data 126 to determine the quantitypicked or placed during an interaction, such as described above.However, in some situations, the same set of weight characteristics maycorrespond to several possible hypotheses. For example, given cans ofapproximately equal weight, placement of two cans of pet food at a firstdistance from an origin may result in the same weight distribution as aplacement of one can at twice that distance from the origin.

The inventory management system 122 may use non-weight data 128, such asimage data, to determine other information about the interaction. Forexample, the image data may be used to determine if motion is present atthe inventory location 114, if the appearance of the inventory location114 has changed indicative of whether an item 104 has been added orremoved, to determine a count of items at the inventory location 114,and so forth.

Hypotheses based on this image data may be generated. For example, thehypotheses may indicate predicted item identifiers and a probabilitythat the predicted item identifier is associated with the interaction.In another example, the hypotheses may indicate a predicted itemquantity of the interaction and a probability that the predicted itemquantity is associated with the interaction.

The inventory management system 122 may be configured to generate,access, or otherwise determine hypotheses having predictedcharacteristics that correspond to measured characteristics observed inthe sensor data 124. Based on the probabilities associated with thehypotheses, a particular hypothesis may be designated a solution, andthe predicted values of that hypothesis may be deemed to reflect theactual interaction.

The process of using the sensor data 124 to generate interaction data isdiscussed in more detail below. For example, FIGS. 9-11 describe variousprocesses for determining a hypothesis based on information derived fromweight data 126 and non-weight data 128.

In some implementations, items 104 may be processed, such as at thereceiving area 106, to generate at least a portion of the item data 136.For example, an item 104 not previously stored by the inventorymanagement system 122 may be photographed by the camera 120(1), weighed,placed on an optical sensor array 120(13) and data about a shadow 132generated, and so forth. Continuing the example, the item data 136generated may include the weight of a single item 104, a center-of-massof the single item 104, an area of the shadow 132, appearance of theitem 104, absorption threshold comprising data indicative oftransparency of the item 104, and so forth.

During configuration of the system 100, the weight distribution of afully laden inventory location 114 may be stored, as well as the weightdistribution of an inventory location 114 that is empty of items 104.

By using the sensor data 124, the inventory management system 122 isable to maintain item data 136 such as inventory levels of a particularitem 104 at a particular inventory location 114, generate billinginformation without manual intervention by a user 116, or provide otherfunctions. For example, the user 116 may pick an item 104 from theinventory location 114. Using the interaction data based on the sensordata 124 and in conjunction with the item data 136, the inventorymanagement system 122 may correctly determine that a quantity of twocans of dog food have been picked and bill the user 116 accordingly forthe sale price of the item 104.

FIG. 2 is a block diagram 200 illustrating additional details of thefacility 102, according to some implementations. The facility 102 may beconnected to one or more networks 202, which in turn connect to one ormore servers 204. The network 202 may include private networks such asan institutional or personal intranet, public networks such as theInternet, or a combination thereof. The network 202 may utilize wiredtechnologies (e.g., wires, fiber optic cables, and so forth), wirelesstechnologies (e.g., radio frequency, infrared, acoustic, optical, and soforth), or other connection technologies. The network 202 isrepresentative of any type of communication network, including one ormore of data networks or voice networks. The network 202 may beimplemented using wired infrastructure (e.g., copper cable, fiber opticcable, and so forth), a wireless infrastructure (e.g., cellular,microwave, satellite, and so forth), or other connection technologies.

The servers 204 may be configured to execute one or more modules orsoftware applications associated with the inventory management system122, and so forth. While the servers 204 are illustrated as being in alocation outside of the facility 102, in other implementations, at leasta portion of the servers 204 may be located at the facility 102. Theservers 204 are discussed in more detail below with regard to FIG. 3.

The users 116, the totes 118, or other objects in the facility 102 maybe equipped with one or more tags 206. The tags 206 may be configured toemit a signal 208. In one implementation, the tag 206 may be a radiofrequency identification (RFID) tag configured to emit an RF signal 208upon activation by an external signal. For example, the external signalmay comprise a radio frequency signal or a magnetic field configured toenergize or activate the RFID tag 206. In another implementation, thetag 206 may comprise a transmitter and a power source configured topower the transmitter. For example, the tag 206 may comprise a BluetoothLow Energy (BLE) transmitter and battery. In other implementations, thetag 206 may use other techniques to indicate presence of the tag 206.For example, an acoustic tag 206 may be configured to generate anultrasonic signal 208, which is detected by corresponding acousticreceivers. In yet another implementation, the tag 206 may be configuredto emit an optical signal 208.

The inventory management system 122 may be configured to use the tags206 for one or more of identification of the object, determining alocation of the object, and so forth. For example, the users 116 maywear tags 206, the totes 118 may have tags 206 affixed, and so forth,which may be read and, based at least in part on signal strength, usedto determine identity and location.

Generally, the inventory management system 122 or other systemsassociated with the facility 102 may include any number and combinationof input components, output components, and servers 204.

The one or more sensors 120 may be arranged at one or more locationswithin the facility 102, on the exterior of the facility 102, and soforth. For example, the sensors 120 may be mounted on or within a floor,wall, at a ceiling, at an inventory location 114, on a tote 118, may becarried or worn by a user 116, and so forth.

The sensors 120 may include one or more cameras 120(1) or other imagingsensors. The one or more cameras 120(1) may include imaging sensorsconfigured to acquire images of a scene. The cameras 120(1) areconfigured to detect light in one or more wavelengths including, but notlimited to, terahertz, infrared, visible, ultraviolet, and so forth. Thecameras 120(1) may comprise charge coupled devices (CCD), complementarymetal oxide semiconductor (CMOS) devices, microbolometers, and so forth.The inventory management system 122 may use image data acquired by thecameras 120(1) during operation of the facility 102. For example, theinventory management system 122 may identify items 104, users 116, totes118, and so forth, based at least in part on their appearance within theimage data acquired by the cameras 120(1). The cameras 120(1) may bemounted in various locations within the facility 102. For example,cameras 120(1) may be mounted overhead, on inventory locations 114, maybe worn or carried by users 116, may be affixed to totes 118, and soforth.

One or more 3D sensors 120(2) may also be included in the sensors 120.The 3D sensors 120(2) are configured to acquire spatial or 3D data, suchas depth information, about objects within a FOV 130 of a sensor 120.The 3D sensors 120(2) may include range cameras, lidar systems, sonarsystems, radar systems, structured light systems, stereo vision systems,optical interferometry systems, and so forth. The inventory managementsystem 122 may use the 3D data acquired by the 3D sensors 120(2) toidentify objects, determine a location of an object in 3D real space,and so forth.

One or more buttons 120(3) may be configured to accept input from theuser 116. The buttons 120(3) may comprise mechanical, capacitive,optical, or other mechanisms. For example, the buttons 120(3) maycomprise mechanical switches configured to accept an applied force froma touch of the user 116 to generate an input signal. The inventorymanagement system 122 may use data from the buttons 120(3) to receiveinformation from the user 116. For example, the tote 118 may beconfigured with a button 120(3) to accept input from the user 116 andsend information indicative of the input to the inventory managementsystem 122.

The sensors 120 may include one or more touch sensors 120(4). The touchsensors 120(4) may use resistive, capacitive, surface capacitance,projected capacitance, mutual capacitance, optical, InterpolatingForce-Sensitive Resistance (IFSR), or other mechanisms to determine theposition of a touch or near-touch. For example, the IFSR may comprise amaterial configured to change electrical resistance responsive to anapplied force. The location within the material of that change inelectrical resistance may indicate the position of the touch. Theinventory management system 122 may use data from the touch sensors120(4) to receive information from the user 116. For example, the touchsensor 120(4) may be integrated with the tote 118 to provide atouchscreen with which the user 116 may select from a menu one or moreparticular items 104 for picking, enter a manual count of items 104 atan inventory location 114, and so forth.

One or more microphones 120(5) may be configured to acquire informationindicative of sound present in the environment. In some implementations,arrays of microphones 120(5) may be used. These arrays may implementbeamforming techniques to provide for directionality of gain. Theinventory management system 122 may use the one or more microphones120(5) to acquire information from acoustic tags 206, accept voice inputfrom the users 116, determine the location of one or more users 116 inthe facility 102, determine ambient noise level, and so forth.

One or more weight sensors 120(6) are configured to measure the weightof a load, such as the item 104, the user 116, the tote 118, and soforth. The weight sensors 120(6) may be configured to measure the weightof the load at one or more of the inventory locations 114, the tote 118,on the floor of the facility 102, and so forth. The weight sensors120(6) may include one or more sensing mechanisms to determine theweight of a load. These sensing mechanisms may include piezoresistivedevices, piezoelectric devices, capacitive devices, electromagneticdevices, optical devices, potentiometric devices, microelectromechanicaldevices, and so forth. The sensing mechanisms of weight sensors 120(6)may operate as transducers that generate one or more signals based on anapplied force, such as that of the load due to gravity. For example, theweight sensor 120(6) may comprise a load cell having a strain gauge anda structural member that deforms slightly when weight is applied. Bymeasuring a change in the electrical characteristic of the strain gauge,such as capacitance or resistance, the weight may be determined. Theinventory management system 122 may use the data acquired by the weightsensors 120(6) to identify an object, determine a change in the quantityof objects, determine a location of an object, maintain shippingrecords, and so forth.

The sensors 120 may include one or more optical sensors 120(7). Theoptical sensors 120(7) may be configured to provide data indicative ofone or more of color or intensity of light impinging thereupon. Forexample, the optical sensor 120(7) may comprise a photodiode andassociated circuitry configured to generate a signal or data indicativeof an incident flux of photons. As described below, the optical sensorarray 120(13) may comprise a plurality of the optical sensors 120(7).For example, the optical sensor array 120(13) may comprise an array ofambient light sensors such as the ISL76683 as provided by IntersilCorporation of Milpitas, Calif., USA, or the MAX44009 as provided byMaxim Integrated, Inc. of San Jose, Calif., USA. In otherimplementations, other optical sensors 120(7) may be used. The opticalsensors 120(7) may be sensitive to one or more of infrared light,visible light, or ultraviolet light. For example, the optical sensors120(7) may be sensitive to infrared light, and infrared light sourcessuch as LEDs may be used to provide illumination. The optical sensors120(7) may include photodiodes, photoresistors, photovoltaic cells,quantum dot photoconductors, bolometers, pyroelectric infrareddetectors, and so forth. For example, the optical sensor 120(7) may usegermanium photodiodes to detect infrared light.

One or more radio frequency identification (RFID) readers 120(8), nearfield communication (NFC) systems, and so forth, may be included assensors 120. For example, the RFID readers 120(8) may be configured toread the RF tags 206. Information acquired by the RFID reader 120(8) maybe used by the inventory management system 122 to identify an objectassociated with the RF tag 206 such as the item 104, the user 116, thetote 118, and so forth. For example, based on information from the RFIDreaders 120(8) detecting the RF tag 206 at different times and RFIDreaders 120(8) having different locations in the facility 102, avelocity of the RF tag 206 may be determined.

One or more RF receivers 120(9) may also be included as sensors 120. Insome implementations, the RF receivers 120(9) may be part of transceiverassemblies. The RF receivers 120(9) may be configured to acquire RFsignals 208 associated with Wi-Fi, Bluetooth, ZigBee, 3G, 4G, LTE, orother wireless data transmission technologies. The RF receivers 120(9)may provide information associated with data transmitted via radiofrequencies, signal strength of RF signals 208, and so forth. Forexample, information from the RF receivers 120(9) may be used by theinventory management system 122 to determine a location of an RF source,such as a communication interface onboard the tote 118 or carried by theuser 116.

The sensors 120 may include one or more accelerometers 120(10), whichmay be worn or carried by the user 116, mounted to the tote 118, and soforth. The accelerometers 120(10) may provide information such as thedirection and magnitude of an imposed acceleration. Data such as rate ofacceleration, determination of changes in direction, speed, and soforth, may be determined using the accelerometers 120(10).

The sensors 120 may also include one or more gyroscopes 120(11). Thegyroscope 120(11) may provide information indicative of rotation of anobject affixed thereto. For example, the tote 118 or other objects maybe equipped with a gyroscope 120(11) to provide data indicative of achange in orientation of the object.

One or more magnetometers 120(12) may be included as sensors 120. Themagnetometer 120(12) may be used to determine an orientation bymeasuring ambient magnetic fields, such as the terrestrial magneticfield. The magnetometer 120(12) may be worn or carried by the user 116,mounted to the tote 118, and so forth. For example, the magnetometer120(12) mounted to the tote 118 may act as a compass and provideinformation indicative of which direction the tote 118 is oriented.

As described above, the optical sensor array 120(13) may comprise one oroptical sensors 120(7). The optical sensors 120(7) may be arranged in aregular, repeating, or periodic two-dimensional arrangement such as agrid. The optical sensor array 120(13) may generate image data.

The sensors 120 may include proximity sensors 120(14) used to determinea presence of an object, such as the user 116, the tote 118, and soforth. The proximity sensors 120(14) may use optical, electrical,ultrasonic, electromagnetic, or other techniques to determine a presenceof an object. In some implementations, the proximity sensors 120(14) mayuse an optical emitter and an optical detector to determine proximity.For example, an optical emitter may emit light, a portion of which maythen be reflected by the object back to the optical detector to providean indication that the object is proximate to the proximity sensor120(14). In other implementations, the proximity sensors 120(14) maycomprise a capacitive proximity sensor 120(14) configured to provide anelectrical field and determine a change in electrical capacitance due topresence or absence of an object within the electrical field.

The proximity sensors 120(14) may be configured to provide sensor data124 indicative of one or more of a presence or absence of an object, adistance to the object, or characteristics of the object. An opticalproximity sensor 120(14) may use time-of-flight (ToF), structured light,interferometry, or other techniques to generate sensor data 124 such asdistance. For example, ToF determines a propagation time (or“round-trip” time) of a pulse of emitted light from an optical emitteror illuminator that is reflected or otherwise returned to an opticaldetector. By dividing the propagation time in half and multiplying theresult by the speed of light in air, the distance to an object may bedetermined. In another implementation, a structured light pattern may beprovided by the optical emitter. A portion of the structured lightpattern may then be detected on the object using an imaging sensor suchas a camera 120(1). Based on an apparent distance between the featuresof the structured light pattern, the distance to the object may becalculated. Other techniques may also be used to determine distance tothe object. In another example, the color of the reflected light may beused to characterize the object, such as skin, clothing, tote 118, andso forth.

In some implementations, a proximity sensor 120(14) may be installed atthe inventory location 114. The non-weight data 128 generated by theproximity sensor 120(14) may be used in conjunction with the weight data126 as described in this disclosure. For example, the optical proximitysensor 120(14) may generate non-weight data 128 indicative of the user116 being within a threshold distance of an inventory location 114.Based on the non-weight data 128, the inventory management system 122may generate activity data indicative of the presence of the user 116.By using the activity data, the inventory management system 122 maydetermine that the weight data 126 is reliable and subsequently usechanges in the weight data 126 to change the item data 136 indicative ofa quantity on hand.

The sensors 120 may include other sensors 120(S) as well. For example,the other sensors 120(S) may include light curtains, ultrasonicrangefinders, thermometers, barometric sensors, hygrometers, biometricinput devices including, but not limited to, fingerprint readers or palmscanners, in-shelf sensors, and so forth. For example, the inventorymanagement system 122 may use information acquired from light curtainsto determine where the user 116 has reached into an inventory location114. The light curtain may comprise one or more pairs of opticalemitters and detectors. An object, such as a hand of the user 116, thatblocks a beam of light sent from the optical emitter to the detectorprovides an indication of the location of the user's 116 hand. Inanother example, in-shelf sensors may generate sensor data 124indicative of a position of an auto-facing unit, proximity of an item104 at the inventory location 114, and so forth.

In some implementations, the camera 120(1) or other sensors 120 mayinclude hardware processors, memory, and other elements configured toperform various functions. For example, the cameras 120(1) may beconfigured to generate image data, send the image data to another devicesuch as the server 204, and so forth.

The facility 102 may include one or more access points 210 configured toestablish one or more wireless networks. The access points 210 may useWi-Fi, NFC, Bluetooth, or other technologies to establish wirelesscommunications between a device and the network 202. The wirelessnetworks allow the devices to communicate with one or more of thesensors 120, the inventory management system 122, the optical sensorarrays 120(13), the tags 206, a communication device of the tote 118, orother devices.

Output devices 212 may also be provided in the facility 102. The outputdevices 212 are configured to generate signals, which may be perceivedby the user 116 or detected by the sensors 120. In some implementations,the output devices 212 may be used to provide illumination of theoptical sensor array 120(13).

Haptic output devices 212(1) are configured to provide a signal thatresults in a tactile sensation to the user 116. The haptic outputdevices 212(1) may use one or more mechanisms such as electricalstimulation or mechanical displacement to provide the signal. Forexample, the haptic output devices 212(1) may be configured to generatea modulated electrical signal, which produces an apparent tactilesensation in one or more fingers of the user 116. In another example,the haptic output devices 212(1) may comprise piezoelectric or rotarymotor devices configured to provide a vibration, which may be felt bythe user 116.

One or more audio output devices 212(2) may be configured to provideacoustic output. The acoustic output includes one or more of infrasonicsound, audible sound, or ultrasonic sound. The audio output devices212(2) may use one or more mechanisms to generate the acoustic output.These mechanisms may include, but are not limited to, the following:voice coils, piezoelectric elements, magnetostrictive elements,electrostatic elements, and so forth. For example, a piezoelectricbuzzer or a speaker may be used to provide acoustic output.

The display devices 212(3) may be configured to provide output, whichmay be seen by the user 116 or detected by a light-sensitive sensor suchas a camera 120(1) or an optical sensor 120(7). In some implementations,the display devices 212(3) may be configured to produce output in one ormore of infrared, visible, or ultraviolet light. The output may bemonochrome or color.

The display devices 212(3) may be emissive, reflective, or both. Anemissive display device 212(3) is configured to emit light duringoperation. For example, the emissive display device 212(3) may producean image using LEDs. In comparison, a reflective display device 212(3)relies on ambient light to present an image. For example, the reflectivedisplay device 212(3) may use an electrophoretic element that emits nolight. Backlights or front lights may be used to illuminate non-emissivedisplay devices 212(3) to provide visibility of the output in conditionswhere the ambient light levels are low.

The display devices 212(3) may include, but are not limited to,micro-electromechanical systems (MEMS), spatial light modulators,electroluminescent displays, quantum dot displays, liquid crystal onsilicon (LCOS) displays, cholesteric displays, interferometric displays,liquid crystal displays (LCDs), electrophoretic displays, and so forth.For example, the display device 212(3) may use a light source and anarray of MEMS-controlled mirrors to selectively direct light from thelight source to produce an image. These display mechanisms may beconfigured to emit light, modulate incident light emitted from anothersource, or both. The display devices 212(3) may operate as panels,projectors, and so forth.

The display devices 212(3) may be configured to present images. Forexample, the display device 212(3) may comprise an addressable display212(3)(1). The addressable display 212(3)(1) may comprise elements thatmay be independently addressable to produce output, such as pixels. Forexample, the addressable display 212(3)(1) may produce an image using atwo-dimensional array of pixels.

In some implementations, the display devices 212(3) may be configured toprovide non-image data, such as text characters, colors, and so forth.For example, an addressable display 212(3)(1) may comprise a segmentedelectrophoretic display device 212(3), segmented LED, and so forth. Theaddressable display 212(3)(1) may be used to present information such asa stock keeping unit (SKU) number, quantity on hand, and so forth. Thedisplay devices 212(3) may also be configurable to vary the color of thesegment, such as using multicolor/multi-wavelength LED segments.

The display devices 212(3) may include image projectors 212(3)(2). Forexample, the image projector 212(3)(2) may be configured to project animage onto objects, illuminate at least a portion of an optical sensorarray 120(13), and so forth. The image may be generated using MEMS,LCOS, lasers, and so forth.

The display devices 212(3) may include a light array 212(3)(3). Thelight array 212(3)(3) may comprise a plurality of discrete emissiveelements configurable to emit light. The discrete emissive elements (orassemblies thereof) may be separated from one another by a distance suchthat, when image data of the light array 212(3)(3) is acquired, oneemissive element may be distinguished from another. For example, thelight array 212(3)(3) may comprise a plurality of infrared LEDsseparated by at least 0.5 centimeters.

Other display devices 212(3)(D) may also be used in conjunction with thefacility 102. The display devices 212(3) may be located at variouspoints within the facility 102. For example, the addressable displays212(3)(1) or the light arrays 212(3)(3) may be located on inventorylocations 114, totes 118, in or on the floor of the facility 102, and soforth.

Other output devices 212(P) may also be present. For example, the otheroutput devices 212(P) may include scent/odor dispensers, documentprinters, 3D printers or fabrication equipment, and so forth.

FIG. 3 illustrates a block diagram 300 of a server 204 configured tosupport operation of the facility 102, according to someimplementations. The server 204 may be physically present at thefacility 102, may be accessible by the network 202, or a combination ofboth. The server 204 does not require end-user knowledge of the physicallocation and configuration of the system that delivers the services.Common expressions associated with the server 204 may include “on-demandcomputing”, “software as a service (SaaS)”, “platform computing”,“network-accessible platform”, “cloud services”, “data centers”, and soforth. Services provided by the server 204 may be distributed across oneor more physical or virtual devices.

One or more power supplies 302 may be configured to provide electricalpower suitable for operating the components in the server 204. The oneor more power supplies 302 may comprise batteries, capacitors, fuelcells, photovoltaic cells, wireless power receivers, conductivecouplings suitable for attachment to an external power source such asprovided by an electric utility, and so forth. The server 204 mayinclude one or more hardware processors 304 (processors) configured toexecute one or more stored instructions. The processors 304 may compriseone or more cores. One or more clocks 306 may provide informationindicative of date, time, ticks, and so forth. For example, theprocessor 304 may use data from the clock 306 to associate a particularinteraction with a particular point in time.

The server 204 may include one or more communication interfaces 308 suchas input/output (I/O) interfaces 310, network interfaces 312, and soforth. The communication interfaces 308 enable the server 204, orcomponents thereof, to communicate with other devices or components. Thecommunication interfaces 308 may include one or more I/O interfaces 310.The I/O interfaces 310 may comprise Inter-Integrated Circuit (I2C),Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) aspromulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s) 310 may couple to one or more I/O devices 314. TheI/O devices 314 may include input devices such as one or more of asensor 120, keyboard, mouse, scanner, and so forth. The I/O devices 314may also include output devices 212 such as one or more of a displaydevice 212(3), printer, audio speakers, and so forth. In someembodiments, the I/O devices 314 may be physically incorporated with theserver 204 or may be externally placed.

The network interfaces 312 may be configured to provide communicationsbetween the server 204 and other devices, such as the totes 118,routers, access points 210, and so forth. The network interfaces 312 mayinclude devices configured to couple to personal area networks (PANs),local area networks (LANs), wide area networks (WANs), and so forth. Forexample, the network interfaces 312 may include devices compatible withEthernet, Wi-Fi, Bluetooth, ZigBee, and so forth.

The server 204 may also include one or more buses or other internalcommunications hardware or software that allow for the transfer of databetween the various modules and components of the server 204.

As shown in FIG. 3, the server 204 includes one or more memories 316.The memory 316 may comprise one or more non-transitory computer-readablestorage media (CRSM). The CRSM may be any one or more of an electronicstorage medium, a magnetic storage medium, an optical storage medium, aquantum storage medium, a mechanical computer storage medium, and soforth. The memory 316 provides storage of computer-readableinstructions, data structures, program modules, and other data for theoperation of the server 204. A few example functional modules are shownstored in the memory 316, although the same functionality mayalternatively be implemented in hardware, firmware, or as a system on achip (SOC).

The memory 316 may include at least one operating system (OS) module318. The OS module 318 is configured to manage hardware resource devicessuch as the I/O interfaces 310, the I/O devices 314, the communicationinterfaces 308, and provide various services to applications or modulesexecuting on the processors 304. The OS module 318 may implement avariant of the FreeBSD operating system as promulgated by the FreeBSDProject; other UNIX or UNIX-like variants; a variation of the Linuxoperating system as promulgated by Linus Torvalds; the Windows operatingsystem from Microsoft Corporation of Redmond, Wash., USA; and so forth.

Also stored in the memory 316 may be a data store 320 and one or more ofthe following modules. These modules may be executed as foregroundapplications, background tasks, daemons, and so forth. The data store320 may use a flat file, database, linked list, tree, executable code,script, or other data structure to store information. In someimplementations, the data store 320, or a portion thereof, may bedistributed across one or more other devices including the servers 204,network attached storage devices, and so forth.

A communication module 322 may be configured to establish communicationswith one or more of the totes 118, sensors 120, display devices 212(3),other servers 204, or other devices. The communications may beauthenticated, encrypted, and so forth.

The memory 316 may store an inventory management module 324. Theinventory management module 324 is configured to provide the inventoryfunctions as described herein with regard to the inventory managementsystem 122. For example, the inventory management module 324 may trackitems 104 between different inventory locations 114, to and from thetotes 118, and so forth.

The inventory management module 324 may include one or more of a dataacquisition module 326 or a processing module 328. The data acquisitionmodule 326 may be configured to acquire and access informationassociated with operation of the facility 102. For example, the dataacquisition module 326 may be configured to acquire sensor data 124 fromone or more of the sensors 120. This information may be stored in thedata store 320 as part of the sensor data 124.

The processing module 328 may be configured to use one or more of sensordata 124, item data 136, partition data 330, threshold data 332, orother information to generate intermediate data 334. One or more of thesensor data 124 or the intermediate data 334 may be used to generateactivity data 336. In some implementations, the activity data 336 may beused to determine one or more hypotheses of the hypotheses data 340.

The threshold data 332 may comprise one or more thresholds associatedwith operation of the processing module 328. For example, the thresholddata 332 may specify a tolerance or accepted extent of deviationpermitted between a hypothesis and the observed sensor data 124. Inanother example, the threshold data 332 may include a thresholdindicating a minimum count of pixels that, if different, designate achange has occurred in image data. The threshold data 332 may includeother thresholds, such as a confidence threshold value, an interactionthreshold period, and so forth. In some implementations, the thresholddata 332 may include constraint data, such as described below that maybe used to constrain the hypotheses data 340.

In some implementations, the threshold data 332 may be based on theerror data 338. For example, the threshold values may be dynamicallyadjusted. A probability density function may be used to determineprobability that a variance in weight is due to sensor error, ratherthan a measured change in load. This probability may be used to set ordefine the threshold value.

Error data 338 may be stored in the data store 320. The error data 338may comprise one or more probability density functions. The probabilitydensity functions describe weight sensor error as a function of weightmeasured. For example, the probability density function may indicatethat a change in weight of 400 g as measured by the weight sensor 120(6)of the inventory location 114 has a probability of 0.01 of occurring asa result of noise or sensor error.

The probability density function may be determined based on theoreticalmodelling, weight data 126 obtained from weight sensors 120(6), or acombination thereof. For example, the weight data 126 may be obtained bypicking and placing objects of known weights onto a representativeinventory location 114 equipped with weight sensors 120(6) and comparingthe known weight to the weight recorded. Different probability densityfunctions, and corresponding error data 338, may be associated withdifferent designs of inventory location 114. For example, an inventorylocation 114 that is configured to store dry items 104 may have adifferent probability density function than an inventory location 114that is configured to store frozen items 104.

In some implementations, the error data 338 may comprise a lookup tablebased on the probability density function. In other implementations, anequation or expression descriptive of the probability density functionmay be used to determine the probability associated with a particularweight.

The processing module 328 may be configured to determine or otherwiseutilize the hypotheses data 340 during operation. The hypotheses data340 may comprise different combinations of values for variables and thecorresponding predicted characteristics based on those different values.The variables may include quantities of items 104, placement of the item104 within the partitioned areas 134 of a particular inventory location114, weight characteristics, and so forth. The weight characteristicsmay include, but are not limited to, position of a center-of-mass (COM),direction of a change in the COM from one time to another, location of aweight change (LWC), and so forth. For example, instead of or inaddition to the weight distribution, the COM of the inventory location114 may be determined using measured weight data 126 and compared with apredicted COM in the hypotheses data 340. In another example, thehypotheses data 340 may include a predicted change in the weightdistribution data. A change in the weight distribution data may beobtained by subtracting weight distribution data associated with a firsttime (or condition) from weight distribution data associated with asecond time (or condition). For example, the predicted weightdistribution change may be based on a change in the quantity of items atone or more of the partitioned areas 134.

In some implementations, the hypotheses data 340 may be constrained. Forexample, the hypotheses data 340 may exclude situations such as asimultaneous pick and place of items 104 in the same interaction,simultaneous removal of items 104 from different partitioned areas 134,interactions involving a minimum or maximum number of items 104, and soforth. Continuing the example, constraint data may be specified whichlimits the hypotheses data 340 to those hypotheses that do not havepredicted quantities that exceed the stowage capacity of the inventorylocation 114, the partitioned area 134, and so forth.

The hypotheses data 340 may be at least partially precomputed. Forexample, given the item data 136 indicating what items 104 are intendedto be stored, weight of the items 104, and so forth, variouspermutations of weight distributions may be generated corresponding todifferent hypotheses.

A first set of hypotheses may be generated using the non-weight data128. For example, image data may be processed using machine visiontechniques to determine a change in appearance at a particularpartitioned area 134. This change in appearance, such as the addition orremoval of an item 104, indicates interaction with the particular item104 associated with that partitioned area 134. Based on thisinformation, corresponding first set of hypotheses data 340 may begenerated that comprises a data structure with information such as:

-   -   Image Hypothesis 1=One item identified as SKU #12345 with a        probability of being true of 0.95.    -   Image Hypothesis 2=Two items identified as SKU #12345 and #67890        with a probability of being true of 0.03.    -   Image Hypothesis 3=Item identified as SKU #88771 with a        probability of being true of 0.02.

The weight data 126 may be used to determine hypotheses data 340 aswell. For example, a second set of hypotheses may be generated thatcomprises a data structure with information such as:

-   -   Weight Hypothesis 1=Quantity of 2 cans of pet food SKU #12345        removed from partitioned area 134(1) exhibits a weight        distribution of 3213 g left and 2214 g right, a change in weight        of 910 g, and a probability of being true of 0.42.    -   Weight Hypothesis 2=Quantity of 1 can of pet food SKU #12345        removed from partitioned area 134(2) and quantity of 2 cans of        pet food SKU #67890 removed from partitioned area 134(1)        exhibits a weight distribution of 2523 g left and 2424 g right,        and a change in weight of 1390 g, and a probability of being        true of 0.14.    -   Weight Hypothesis 3=Quantity of 1 box of pet food SKU #88771        removed from partitioned area 134(3) exhibits a weight        distribution of 3523 g left and 1917 g right, and a change in        weight of 897 g, and a probability of being true of 0.44.

A solution may be determined from a set of one or more hypotheses basedon a confidence value. The confidence value may be determined based on adifference or ratio between probability values of two or more hypothesesin the hypotheses data 340. In one implementation, the confidence valuemay be calculated as a difference between the probabilities of the toptwo hypotheses in a set of hypotheses, when ranked by probability. Forexample, the hypotheses data 340 may include a first set of hypothesesbased on non-weight data 128, such as image data, from the example abovethat are ranked in descending order of probability in Table 1.

TABLE 1 Hypothesis Probability Rank Image Hypothesis 1 0.95 First ImageHypothesis 2 0.03 Second Image Hypothesis 3 0.02 Third

The confidence value may be determined as a difference between theprobabilities of the first and second ranked hypotheses. In thisexample, the confidence value for the first ranked Image Hypothesis 1 is0.95−0.03=0.92. In other implementations, the confidence value maycomprise a ratio.

Continuing the example above, the hypotheses data 340 may include thesecond set of hypotheses based on weight data 126 from the example abovethat are ranked in descending order of probability in Table 2.

TABLE 2 Hypothesis Probability Rank Weight Hypothesis 3 0.44 FirstWeight Hypothesis 1 0.42 Second Weight Hypothesis 2 0.14 Third

In this example, the confidence value for the first ranked ImageHypothesis 1 is 0.44−0.42=0.02.

The confidence value may be compared to a confidence threshold value.For example, the confidence threshold value may be 0.60. Hypotheses withconfidence values below the threshold may be deemed to have a “lowconfidence” while those at or above the threshold may be deemed to have“high confidence”. The first ranked hypothesis that has a confidencevalue above the confidence threshold value may be deemed a solution.

Low confidence hypotheses, or the set of hypotheses of which they are apart, may be combined or aggregated with one or more other sets ofhypotheses to produce one or more high confidence hypotheses. Forexample, the weight hypotheses of the second set of hypotheses provideinformation about what items 104 and the quantities of the interaction,but no “high confidence” hypotheses are available. To improve theconfidence value, additional sets of hypotheses may be combined. Theseadditional sets of hypotheses may be based on sensor data 124 from othersensors 120, from output resulting from different processing of the samesensor data 124, and so forth. Continuing the example above, the firstset of hypotheses may be generated based on the non-weight data 128while the second set of hypotheses are generated based on the weightdata 126. At least a portion of the first set of hypotheses and thesecond set of hypotheses may be combined.

The individual hypotheses from different sets may be combined and anaggregate probability may be determined. For example, the first set ofhypotheses and the second set of hypotheses may be combined to form athird set of hypotheses. In one implementation, a combined probabilitythat is the product of the probabilities of pairs of hypotheses from thefirst set and the second set may be calculated. For example, theaggregate probability for the hypotheses may be:

TABLE 2 Image Weight Combined Hypothesis Probability ProbabilityProbability Rank Combined 1 0.95 0.42 0.399 First Combined 3 0.02 0.440.009 Second Combined 2 0.03 0.14 0.004 Third

The combined hypothesis that has the greatest combined probability maybe deemed to be the solution. In this example, the solution describestwo cans of the item 104 having SKU #12345 being removed from thepartitioned area 134(1). The solution may then be used to generateinteraction data 342 that describes the removal of two cans of pet foodhaving SKU #12345 from the partitioned area 134(1). The quantity on handat that partitioned area 134(1) may be decreased accordingly, and thequantity determined to be in possession of the user 116 may be increasedaccordingly.

In other implementations, instead of or in addition to theprobabilities, the measured change in characteristics such as weightdistribution data may be compared with the predicted change incharacteristics to determine a solution. For example, the determinationof the hypotheses may be based on the absolute value of the differencebetween the measured weight distribution and the predicted weightdistribution exceeding a threshold value.

In some implementations, the error data 338 may be used to determine aparticular hypothesis in the hypotheses data 340. For example, the errordata 338 may be used to determine a probability that a variance betweenthe predicted weight distribution and the measured weight distributionis due to a sensor error. A high probability may indicate that theparticular variance is the result of a sensor error and not a change inload. The processing module 328 may generate a score based on theprobability of the error data 338.

The processing module 328 may generate reliability data 344 indicativeof the reliability of weight data 126 based on non-weight data 128. Forexample, the reliability data 344 may indicate that the weight data 126is unreliable when no activity at the inventory location 114 isdetected. As a result, spurious data is not processed, preventingincorrect changes in quantity on hand, or other effects.

Operation of the processing module 328 and the various data involvedincluding the intermediate data 334, activity data 336, hypotheses data340, reliability data 344, and so forth, is discussed in more detailbelow.

Processing of the sensor data 124, intermediate data 334, or other datamay be performed by the processing module 328 or other modulesimplementing, at least in part, one or more of the following tools ortechniques. In one implementation, the OpenCV library as developed byIntel Corporation of Santa Clara, Calif., USA; Willow Garage of MenloPark, Calif., USA; and Itseez of Nizhny Novgorod, Russia, withinformation available at www.opencv.org may be used. In anotherimplementation, functions available in the OKAO machine vision libraryas promulgated by Omron Corporation of Kyoto, Japan, may be used toprocess the sensor data 124.

Techniques such as artificial neural networks (ANN), active appearancemodels (AAM), active shape models (ASM), principal component analysis(PCA), and so forth, may also be used to process the sensor data 124,the intermediate data 334, the activity data 336, or other data. Forexample, the ANN may be a trained using a supervised learning algorithmsuch that object identifiers are associated with images of particularobjects within training images provided to the ANN. Once trained, theANN may be provided with the sensor data 124 such as the image data froma camera 120(1), and may provide, as output, the object identifier. Forexample, the ANN may be trained to recognize a particular item 104 atthe inventory location 114, determine a count of items 104 in aparticular partitioned area 134, and so forth.

Other modules 346 may also be present in the memory 316, as well asother data 348, in the data store 320. For example, the other modules346 may include an accounting module while the other data 348 mayinclude billing data. The accounting module may be configured to assesscharges to accounts associated with particular users 116 or otherentities, while the billing data may include information such as paymentaccount numbers. In some implementations, the other data 348 may includephysical configuration data. The physical configuration data mayindicate dimensions of an inventory location 114, placement of weightsensors 120(6), tare weight of the inventory location 114, and so forth.

FIG. 4 is a block diagram 400 of additional data that may be used by theserver 204 to support operation of the facility 102, according to someimplementations. This data may be stored at least in part by the datastore 320. As described above with regard to FIG. 3, the inventorymanagement module 324 may use the sensor data 124 to generate otherinformation such as interaction data 342 indicative of what item 104 auser 116 has interacted with.

The processing module 328 may access item data 136. The item data 136may include an item identifier 402. The item identifier 402 may be usedto distinguish one item 104 from another. For example, the itemidentifier 402 may include a SKU, Universal Product Code (UPC) number,and so forth. The items 104 that are of the same type may be referred toby the same item identifier 402. For example, 450 g cans of dog food maybe represented by the item identifier 402 value of “9811901181”. Inother implementations, non-fungible items 104 may each be provided witha unique item identifier allowing each item 104 to be distinguished fromone another.

The item data 136 may include one or more of item weight data 404,geometry data 406, or absorption threshold data 408. The item weightdata 404 comprises information indicative of a weight of the item 104,package, kit, or other grouping considered to be a single item 104. Thegeometry data 406 may comprise information indicative of an area of ashadow 132 of the item 104, within an image of the item 104 acquired bya camera 120(1), and so forth. For example, the geometry data 406 maycomprise an area as measured in pixels, square centimeters, and soforth. The geometry data 406 may be for a single item 104 or a packageor kit of objects considered to be a single item 104.

The geometry data 406 may also comprise information indicative of theshape of the item 104, such as in the shadow 132, an image acquired froma camera 120(1), and so forth. The geometry data 406 may compriseinformation indicative of one or more contours of the item 104. Forexample, the geometry data 406 may comprise information indicating thatthe shadow 132 of the item 104 is a rectangle measuring three pixels byseven pixels.

The absorption threshold data 408 provides information indicative of howtransparent the item 104 is to the light detected by the optical sensorarray 120(13). For example, the absorption threshold data 408 maycomprise a 75^(th) percentile value of the light intensity values of thepixels that are within a contour of the shadow 132 of the single item104. The absorption threshold data 408 may be used to generateintermediate data 334, such as binary image data described below.

The item data 136 may include one or more inventory location identifiers(IDs) 410. The inventory location IDs 410 are indicative of a particularinventory location 114 that is designated for stowage of the item 104.The item data 136 may also include quantity data 412. The quantity data412 may comprise a count or value indicative of a number of items 104.The count may be a measured or estimated value. The quantity data 412may be associated with a particular inventory location ID 410, such aswhen the same item 104 is stowed at several different inventorylocations 114 or in a particular partitioned area 134. In otherimplementations, the item data 136 may include other data, such as animage of the item 104.

The partition data 330 may include one or more of a partition identifier414, inventory location ID 410, a sensor identifier 416, or partitioncoordinates 418. As described above, a single inventory location 114with an optical sensor array 120(13) may stow several different kinds ofitems 104, with each item 104 being associated with a different itemidentifier 402. For example, the optical sensor array 120(13) may beincorporated into a shelf of the inventory location 114. Three differentitems 104(1), 104(2), and 104(3) may be stored on that same shelfserviced by a single optical sensor array 120(13), with each item 104 ina partitioned area 134(1), 134(2), 134(3), respectively.

The partition identifier 414 comprises data indicative of a particularpartitioned area 134. The partition identifier 414 may be unique in thecontext of a particular inventory location 114, aisle 112, facility 102,or globally across multiple facilities 102. The inventory location ID410 included in the partition data 330 may associate the particularpartition identifier 414 with a particular inventory location 114. Theparticular inventory location 114 may then be associated with the itemidentifier 402 indicative of the items 104 stored therein.

The sensor identifier 416 comprises data indicative of a particularsensor 120, such as a certain camera 120(1), optical sensor array120(13), proximity sensor 120(14), and so forth. The sensor identifier416 may be unique in the context of a particular inventory location 114,aisle 112, facility 102, or globally across multiple facilities 102.

The partition coordinates 418 may specify an area that encompasses thepartitioned area 134. For example, the partition coordinates 418 mayspecify the location in space with respect to an origin of two or moreof corners of a rectangular partitioned area 134, a distance relative toan origin, and so forth.

The sensor data 124 may be generated or acquired by one or more sensors120. The sensor data 124 may include one or more of the sensoridentifiers 416, a timestamp 420, weight data 126, or non-weight data128. The timestamp 420 may comprise information indicative of a timewhen the sensor data 124 was acquired. For example, the timestamp 420may be based at least in part on time data obtained from a clock onboardthe sensor 120, by the clock 306 on the server 204, and so forth. Theinventory management module 324 may use the timestamp 420 to associateweight data 126 with non-weight data 128 and a corresponding time.

As described above, the sensor data 124 may be broadly categorized ascomprising weight data 126 and non-weight data 128. For example, theweight data 126 may comprise information obtained from one or more theweight sensors 120(6). Conversely, the non-weight data 128 may compriseinformation obtained from the sensors 120 other than the weight sensors120(6). For example, the non-weight data 128 may be obtained from thecameras 120(1), 3D sensors 120(2), optical sensors 120(7), opticalsensor arrays 120(13), proximity sensors 120(14), and so forth.

In some implementations, the non-weight data 128 may comprise image data422. The image data 422 may be obtained from one or more sensors 120,such as a camera 120(1), a 3D sensor 120(2), or optical sensor array120(13). The image data 422 may comprise one or more pixels 424. In oneimplementation, the image data 422 may be represented as atwo-dimensional matrix.

In the implementation where the image data 422 is provided by an opticalsensor array 120(13), the pixels 424 may comprise data acquired from oneor more of the optical sensors 120(7). For example, a single opticalsensor 120(7) may be represented by a single pixel 424. Each pixel 424may include information indicative of a light intensity value 426. Thelight intensity value 426 provides information indicative of a flux oflight impinging upon the optical sensor 120(7) at a particular time. Forexample, the light intensity value 426 may comprise an 8 or 16 bit valueproduced by the optical sensor 120(7). The pixel 424 may also includeinformation indicative of a coordinate 428 or relative position of thepixel 424 with respect to other pixels 424 or an origin point. Forexample, the coordinates 428 may indicate that a particular pixel 424 isat an intersection of a particular row and column. The coordinates 428may express a relative position within the two-dimensional arrangementof the optical sensor array 120(13).

In some implementations, the pixels 424 may also include color orspectral data. For example, each pixel 424 may have a plurality of lightintensity values 426, with each of the light intensity values 426indicative of an intensity of a different wavelength or range ofwavelengths of light.

The sensor data 124 may include other data 430. For example, other data430 may include information indicative of operational status of thesensor 120, error messages associated with the sensor 120, and so forth.

The processing module 328 may access the item data 136, the partitiondata 330, and the sensor data 124 to generate intermediate data 334. Forexample, the processing module 328 may access threshold data 332 and theimage data 422 to generate binary image data 432.

The threshold data 332 may include a binary image threshold value usedto distinguish whether a pixel 424 in the resulting binary image data432 will be designated as a binary “0” value or binary “1” value. Forexample, the binary image data 432 may be generated by comparing thelight intensity value 426 of each pixel 424 with a threshold value. Inthis example, the threshold value may be an 8-bit value of “50”. Thepixels 424 having light intensity value 426 below 50 may result in apixel 424 in the binary image data 432 having a binary value of “1”. Inother implementations, the opposite values may be used, such as valuesbelow the threshold value being assigned a binary value of “0”.

By thresholding in this fashion, the resulting binary image data 432 maybe more easily processed to determine characteristics such as shape,area, perimeter, edges or contours. For example, the OpenCV function“threshold” may be used to generate the binary image data 432. In otherimplementations, other thresholding techniques may be used.

The processing module 328 may be configured to generate contour datausing the binary image data 432. The contour data may provideinformation indicative of a shape having a closed or complete perimeter.In some implementations, the contour data may be indicative of a curveor open perimeter. For example, an edge appearing in the shadow 132 maybe incomplete. This may occur due to an optical anomaly, erroneousreading by an optical sensor 120(7), and so forth. The processing module328 may be configured to close an open perimeter.

The contour data may comprise the coordinates 428 of the pixels 424within the binary image data 432 having a binary value of “1” or “0”. Inother implementations, the contour data may comprise a vector value,matrix of values, or other information representative of the perimeterof a shadow 132. For example, the OpenCV function “FindContours” may beused to generate the contour data. Other functions may be used togenerate the contour data. For example, the OpenCV function“contourArea” may be used to process the binary image data 432 togenerate the geometry data 406. The geometry data 406 may be indicatedin terms of pixels 424, units of linear measurement, and so forth.

In some implementations, the binary image data 432 may be furtherprocessed to reduce noise, simplify later processing, and so forth. Forexample, an erosion function may be applied to the binary image data432. In one implementation, where the contour is represented by binary“1”s in the binary image data 432, the erosion function may beconfigured to set to a value of “0” those pixels 424 adjacent to, butnot part of, a contour. For example, the OpenCV function “erode” may beused. In some implementations, the erosion may use a 1 pixelneighborhood boundary. Second binary image data 432 may be generated asa result of this processing, or the original binary image data 432 maybe modified.

The intermediate data 334 may also comprise differential data 434. Thedifferential data 434 may result from a comparison between image data422 at different times and may be indicative of a change or differencewhen one occurs. The image data 422 may be obtained from the camera120(1), the 3D sensor 120(2), the optical sensor array 120(13), or othersensors 120(S). For example, the differential data 434 may comprise avalue indicating that the particular image data 422 has changed relativeto earlier image data 422. In some implementations, the differentialdata 434 may simplify processing by removing “background” such as thoseitems 104 that were left untouched. The differential data 434 may beused to determine if a change has occurred between a first time and asecond time. For example, the differential data 434 may be generatedusing image data 422 from before and after an event.

The differential data 434 may comprise one or more differential images.In one implementation, the differential images may result fromcomparison of one binary image with another, one frame of image data 422with another, and so forth. For example, a first image sensor data124(1) may be subtracted from a second image sensor data 124(2). Thesubtraction may include subtracting the light intensity values 426 of apixel 424 in the first image sensor data 124(1) from a correspondingpixel 424 at the same coordinates in the second image sensor data124(2). The subtraction may be accomplished using the “cvSub( )”function of OpenCV.

The extent of change between the first image data 422(1) and the secondimage data 422(2) may be quantified by assessment of the differentialimage. In one implementation, pixels 424 in the differential imagehaving a non-zero light intensity value 426 are those that exhibited achange. The pixels 424 in the differential image may be counted, acontour determined from the pixels 424, and so forth.

The processing module 328 may use the differential data 434 to determinea state of the image data 422 at a particular time. The image data 422may be deemed to be “stable” when the number of changes for a particularperiod of time is below a threshold value in the threshold data 332.Similarly, the image data 422 may be deemed to be “unstable” when thenumber of changes for the particular period of time is greater than orequal to the threshold value. For example, the threshold value mayspecify a number of non-zero pixels. Continuing the example, where thethreshold value is 1, differential images having one or fewer pixels 424may be deemed to indicate a stable state of the latest image data 422obtained. As described below, information about whether the image data422 is in a stable or unstable state may be used to distinguish ameasured interaction from noise or a spurious event.

The processing module 328 may also generate intermediate data 334 suchas an interaction duration. The interaction duration may indicate alength of time the image data 422 was deemed to be in an unstable state.For example, the interaction duration may be the time elapsed betweenthe end of a first stable state and the beginning of a second stablestate. In some implementations, the processing module 328 may use aninteraction threshold period stored in the threshold data 332 to preventtransient signals from generating interaction data 342. For example, theinteraction threshold period may be 500 milliseconds (ms). Image data422 for which the unstable state is greater than 500 ms may be processedand subsequently result in the generation of interaction data 342. Incomparison, image data 422 having an unstable state that is less than500 ms in duration may be disregarded as noise or a spurious event.

Intermediate data 334 may also include weight characteristic data 436.The weight characteristic data 436 may be generated by processing theweight data 126. The weight characteristic data 436 may include weightchange data 438, weight distribution data 440, location of weight changedata 442, or other data, such as weight at a particular instant in time.

The weight change data 438 is indicative of a change in weight measuredby one or more of the weight sensors 120(6) from a first time to asecond time. For example, calculation of the weight change data 438 maycomprise subtracting a first weight obtained at the first time from asecond weight obtained at the second time. In some implementations, theinventory management module 324 may determine the weight change data438. In other implementations, the determination of the weight changedata 438 may be performed at least partially onboard the weight sensor120(6) or an associated device such as a controller. In someimplementations, the weight change data 438 may include informationindicative of noise in the weight data 126, variability of the weightdata 126, estimated reliability of the weight data 126, and so forth.

The weight distribution data 440 may provide data indicative of weightdistribution a particular time. The weight distribution data 440 may beexpressed as a measured weight at a particular weight sensor 120(6), aratio or percentage of weight on a weight sensor 120(6), and so forth.For example, the weight distribution data 440 may be expressed as “3213g left, 2214 g right”, “0.59 left, 0.41 right”, and so forth. In someimplementations, the inventory management module 324 may determine theweight distribution data 440. In other implementations, thedetermination of the weight distribution data 440 may be performed atleast partially onboard the weight sensor 120(6) or an associated devicesuch as a controller. In some implementations, the weight distributiondata 440 may indicate a change in the weight distribution from a firsttime to a second time. The weight distribution data 440 may be expressedas a weight associated with one or more of the weight sensors 120(6).For example, the weight distribution data 440 for a configuration inwhich a rectangular shelf has a weight sensor 120(6) at each of-the fourcorners may have weight distribution data 440 corresponding to each ofthe corners. In another example, data from weight sensors 120(6) may becombined, such as to provide a weight measured at a left side of theinventory location 114 and a weight measured at a right side of theinventory location 114. In some implementations, the inventorymanagement module 324 may determine the weight distribution data 440. Inother implementations, the determination of the weight distribution data440 may be performed at least partially on board the weight sensor120(6) or an associated device such as a controller. The weightdistribution data 440 may provide data indicative of center-of-mass(COM) at a particular time. For example, the weight distribution data440 may indicate a COM, change in the COM from a first time to a secondtime, and so forth.

A variety of techniques may be used to calculate the COM orcenter-of-gravity. The COM may be described as a point in space at whichweighted position vectors relative to the point sum to zero. Forexample, the COM of a uniform sphere is a point in the center of thesphere. In another example, the COM of a toroid is a point in the centerof the toroid. A variety of techniques may be used to calculate the COM.Consider a simple system having two masses m1 and m2, arranged along asingle axis “x” at positions x1 and x2, respectively. The position ofeach mass is given as a distance “x” relative to an origin. The COM maybe expressed by the equation:x=((m1*x1)+(m2x2))/(m1+m2)   Equation 1

The physical characteristics of the inventory location 114, placement ofthe weight sensors 120(6) at the inventory location 114, physicalposition of the partitioned area 134 relative to the inventory location114, quantity and weight of the items 104 at the respective partitionedareas 134, and so forth, may be known. For example, given the physicaldesign of the inventory location 114, it may be known that a weightsensor 120(6) is positioned at each of the four corners of a shelf, andthat the shelf has a particular length and width. Continuing theexample, the physical coordinates corresponding to the partitioned area134 on that shelf are known. Using this information, as well as the itemdata 136, weight characteristic data 436 may be generated for aninventory location 114 before, during, or after an interaction.

The location of weight change (LWC) data 442 provides informationindicative of the location, with respect to the inventory location 114,at which a weight change has taken place. For example, the LWC data 442may indicate that a weight change has taken place at 15 cm from theorigin of the inventory location 114. The LWC data 442 may be determinedusing the weight data 126. For example, the LWC data 442 may becalculated from the weight distribution data 440.

In some implementations, the LWC data 442 may be expressed as a vectorvalue having a direction and a magnitude. For example, the LWC data 442may comprise a vector having a first endpoint at an origin of theinventory location 114 and a second endpoint at the location of theweight change.

In one implementation, the LWC data 442 may be determined as follows.Assume a situation wherein the inventory location 114 comprises a shelfhaving a width “a”, a left weight sensor located at a distance “b” fromthe left edge of the shelf and right weight sensor located at a distance“b” from the right edge of the shelf. The weight measured by the leftweight sensor is “w1” and the weight measured by the right weight sensoris “w2”. A distance “LWC” indicative of the location of weight changefrom an origin at the leftmost edge of the shelf may be calculated tothe center-of-mass of an individual item 104 that has been added orremoved in an interaction using the following equation:LWC=w2*(a−2b)/(w2+w1)+b   Equation 2

The weight change corresponding to the interaction may be calculated as:Total weight change=w1+w2   Equation 3

During operation, the weight data 126 may be “tared” or zeroed out whilethe load on the platform measured by the weight sensors 120(6) is in astable state. Subsequent changes in the weight data 126 may be used toproduce the weight distribution data 440. For example, the inventorylocation 114 when fully loaded may have a total weight of 15 kg. Theprocessing module 328 may “tare” these values, such that the weight isread to be “0 kg”. A subsequent interaction such as a removal of twoitems 104 may result in a total weight change of 905 grams, with aweight distribution of 850 g on the left and 55 g on the right. Given ashelf width “a” of 1 m and the distance “b” of 0.1 m, the LWC is at0.148 meters from the origin at the leftmost edge of the shelf.

The processing module 328 or other modules may transform the weightcharacteristic data 436 from one form to another. For example, the LWCdata 442 may be determined using the weight distribution data 440.Similarly, the LWC data 442 may be used to derive a COM.

The inventory management module 324 may be configured to generateactivity data 336, based at least in part on non-weight data 128. Theactivity data 336 provides information indicative of an activity, orlack thereof, at the inventory location 114. The activity data 336 mayinclude one or more of location data 444, motion data 446, duration data448, user identifier 450, or other data 452. In some implementations,the activity data 336 may be generated at least in part using theintermediate data 334. For example, the motion data 446 may be generatedusing differential data 434 obtained from a plurality of images.

The location data 444 provides information indicative of a particularposition or partitioned area 134 that the activity is associated with.For example, a shadow 132 detected by an optical sensor array 120(13)beneath the partitioned area 134(1) may be processed to generatelocation data 444 indicative of the partitioned area 134(1). In someimplementations, the location data 444 may be generated based on thephysical configuration data of the facility 102. For example, given aknown placement of the camera 120(1) above the partitioned area 134, andthe FOV 130 of that camera 120(1) being directed toward the partitionedarea 134, the image data 422 obtained from the camera 120(1) isassociated with that particular location.

The motion data 446 may comprise information indicative of motion of oneor more objects within the facility 102, particularly with regard to theinventory location 114, partitioned area 134, or other particularpointer area. For example, the motion data 446 may indicate that anobject is approaching the inventory location 114. In one implementation,the motion data 446 may be determined at least in part by the image data422 acquired by one or more of the cameras 120(1).

The duration data 448 provides information indicative of the duration ofan activity. For example, the duration data 448 may provide informationabout how long the hand of the user 116 remained within the FOV 130 inthe image data 422. For example, the duration data 448 may be determinedbased on the length of time the differential data 434 was in theunstable state.

The user identifier 450 provides information indicative of a particularuser 116. For example, the user identifier 450 may comprise an accountnumber, account name, key value, serial number, and so forth, associatedwith a particular user 116 or user account. The processing module 328may be configured to determine the user identifier 450. For example, theprocessing module 328 may use facial recognition techniques such asavailable in OpenCV, Itseez, OKAO, and so forth, as described above torecognize a particular user 116 and associate the corresponding useridentifier 450 with that user 116. In other implementations, the useridentifier 450 may be determined by other sensors 120, such as the RFIDreader 120(8) reading a tag 206 carried by the user 116.

The processing module 328 may generate interaction data 342 using theintermediate data 334, activity data 336, and so forth. The interactiondata 342 may comprise information about one or more of the items 104that may be undergoing some change in response to one or more events,such as movement from the inventory location 114 to the tote 118.

The interaction data 342 may comprise one or more of an interaction type454, a partition identifier 414, an item identifier 402, a quantitychange 456, and so forth. For example, differential data 434 such as thelocation of pixels 424 in a differential image produced from image data422 may be used to determine a hand of the user 116 is moving in an areacorresponding to the partitioned area 134.

The interaction type 454 may provide information about whether theinteraction is determined to be a pick, place, touch, pick and place,and so forth. The processing module 328 may use the intermediate data334 or other information such as the sensor data 124, the activity data336, and so forth, to determine the interaction type 454. For example,weight characteristic data 436 may be generated from the weight data126. Based on activity data 336 indicative of motion at the inventorylocation 114, the processing module 328 may determine that the weightdata 126 is reliable. This reliable data may be used for otherprocessing, such as determining the probabilities for the hypotheses asdescribed above.

The partition identifier 414 may indicate the particular partition data330 corresponding to the partitioned area 134 associated with the event.For example, the event may comprise differential data 434 indicatingmotion at the partitioned area 134, a change to the partitioned area 134(such as from addition or removal of an item 104), and so forth. Usingthe partition identifier 414, the item identifier 402 may be determined.For example, a particular portion of the FOV 130 may be associated witha particular partitioned area 134 and the item 104 stowed thereby. As aresult, changes in the image data 422 may be associated with that itemidentifier 402.

The item identifier 402 specifies the item 104 implicated by theinteraction. For example, the item identifier 402 may indicate the item104 that was picked, placed, touched, and so forth. In someimplementations, the item identifier 402 may be determined, at least inpart, by the sensor data 124. For example, based on the differentialdata 434 the partition identifier 414 may be determined. From thepartition identifier 414, an associated item identifier 402 may bedetermined. Using the known weight of the items 104 as stored in theitem data 136, the weight characteristic data 436 may be used to provideinformation about the quantity change of the items 104.

The quantity change 456 provides information indicative of a change inthe quantity of the item 104 resulting from the interaction. Forexample, the quantity change 456 may indicate a value of “−1” when asingle item 104 is picked from the inventory location 114, or a value of“+3” when three items 104 are placed to the inventory location 114.

In one implementation, the processing module 328 may generate otherinformation about the items 104 stowed at the inventory location 114.For example, the interaction data 342 may be analyzed to determine if auser 116 such as a person tasked with restocking the inventory location114 is rotating stock such that old stock is brought to the front whilenew stock is placed behind.

In another implementation, the processing module 328 may generateinformation indicating that an item 104 has been misplaced in anincorrect partitioned area 134. For example, the weight characteristicdata 436 may be compared to item weight data 404. Based on a mismatchbetween the weight characteristic data 436 and the item weight data 404,it may be determined an item 104 has been stowed in the wrongpartitioned area 134.

FIG. 5 illustrates a side view 500 of an inventory location 114 thatincludes various sensors 120, according to some implementations. In thisillustration, the inventory location 114 comprises a shelf 502 on arack.

One or more weight sensors 120(6) may be used to obtain weight data 126from a platform, such as the shelf 502. In this illustration, the weightsensors 120(6) are arranged at the corners of the shelf 502. In anotherimplementation, the weight sensors 120(6) may be mounted on attachmentpoints that affix the shelf 502 to the rack. For example, the bracketsupporting the shelf 502 may include a strain gauge configured for useas a weight sensor 120(6).

Above the shelf 502 may be a light source 504 configured to emit light506. The light source 504 may comprise one or more LEDs, quantum dots,electroluminescent devices, incandescent lamps, fluorescent lamps, andso forth. The light source 504 may be configured to emit light 506 inone or more wavelengths including, but not limited to, infrared,visible, or ultraviolet. In some implementations, to reduce dazzling theeyes of the user 116, the light source 504 may be configured to emitinfrared light 506.

The light source 504 emits light 506 that is detectable by at least aportion of the optical sensors 120(7) in the optical sensor array120(13). In some implementations, the light source 504 may be locatedelsewhere with respect to the optical sensor array 120(13). For example,the light source 504 may comprise an overhead light fixture thatprovides general illumination to the inventory location 114.

The shelf 502 may incorporate the optical sensor array 120(13) asillustrated in FIG. 5. For example, the shelf 502 may comprise astructure such as a piece of glass or plastic that is transparent to thewavelengths of light 506. The items 104 may rest upon the structure, asillustrated here, or may hang above the structure, such as from a peg orarm. In some implementations, items 104 may be stacked one atop another,such as shown here-with stacked cans of pet food.

As a result of the light 506 impinging upon the item 104, a shadow 132is cast upon at least a portion of the optical sensor array 120(13). Theintensity of light within the shadow 132 may be dependent upon thetransparency of the item 104. For example, a clear glass bottle holdingwater may cast a light shadow 132, while an opaque black plastic bottlemay cast a very dark shadow 132. During an interaction, the shadow 132may also be cast, at least in part, by another object such as a hand 508of the user 116.

The optical sensor array 120(13) is configured to provide image data 422to the inventory management module 324. The image data 422 may then beprocessed by the processing module 328 to generate the interaction data342, such as which of the partitioned areas 134 held the item 104 theuser 116 interacted with.

The light source 504 may be configurable to modulate the light 506. Thelight 506 may be modulated such that the optical sensor array 120(13) isable to filter out or disregard other light sources 504 and obtain imagedata 422 based on the light 506 coming from the known position of thelight source 504. Modulation of light 506 may include, but is notlimited to, carrier modulation, amplitude shift keying, pulse positionmodulation, Manchester encoding, and so forth. The optical sensor array120(13) may be configured to process the data from the optical sensors120(7) to generate light intensity values 426 for the light 506 havingthe predetermined modulation. For example, data values associated withnon-modulated light may be disregarded or filtered out.

In another implementation, operation of the light source 504 and theoptical sensor array 120(13) may be time synchronized. For example, thelight source 504 may be configured to emit light 506 at a particulartime and for a particular duration, such as 50 ms. The optical sensorarray 120(13) may be configured to acquire data from the optical sensors120(7) while the light source 504 is emitting light 506. In someimplementations, first image data 422(1) acquired while the light source504 is active may be compared with second image data 422(2) acquiredwhile the light source 504 is inactive. A comparison may be made betweenthe first image data 422(1) and the second image data 422(2) to filterout or otherwise calibrate the system for ambient light.

In still another implementation, the light source 504 may be included inthe optical sensor array 120(13). For example, the light sources 504 maycomprise LEDs that are configured to emit light 506 toward where theitem 104 may be stowed. The light 506 may be reflected from an objectsuch as the hand 508, the item 104, and so forth. The reflected lightmay be detected by one or more of the optical sensors 120(7). In someimplementations, the light 506 may be distributed from the light source504 using an optical waveguide, fiber optic fibers, or other features.

In yet another implementation, the optical sensor array 120(13) maycomprise a plurality of proximity sensors 120(14). The proximity sensors120(14) may use data indicative of proximity of an object such as theitem 104 to generate the image data 422. For example, an array ofcapacitive proximity sensors 120(14) may be used to generate the imagedata 422.

One or more cameras 120(1) may also be positioned at one or more of on,in the, or around the inventory location 114. For example, cameras120(1) may be arranged such that their FOV 130 looks on a shelf 502. Thecameras 120(1) may be arranged at a front edge of the inventory location114 such as closest to the aisle 112 during operation, at the back edgeof the inventory location 114, both at the front and back edges of theinventory location 114, overhead the inventory location 114, and soforth.

FIG. 6 is an illustration 600 of the optical sensor array 120(13),according to some implementations. In this illustration, a top view 602and a side view 604 are presented.

As shown by the top view 602, the optical sensor array 120(13) maycomprise a plurality of optical sensors 120(7). The optical sensors120(7) may be arranged in a two-dimensional arrangement, such as thegrid arrangement depicted here. The arrangement shown here comprises anarray with an inter-sensor distance 606 that is approximately the sameas measured along the X and Y axes. For example, the inter-sensordistance 606 may be at least 5 millimeters (mm) between the centers orthe edges of the optical sensors 120(7). In some implementations, suchas described below with regard to FIG. 8, the inter-sensor distance 606may be representative of a distance between optical elements 612.

In other implementations, other arrangements of the optical sensors120(7) may be used. For example, the arrangement may comprise atriangular space filling array with an optical sensor 120(7) located ateach vertex.

The distribution or arrangement of the optical sensors 120(7) may beasymmetrical. In one implementation, the inter-sensor distance 606 maybe varied. For example, a central region of the optical sensor array120(13) may be sparsely populated with optical sensors 120(7) such thatthe inter-sensor distance 606 along the X and Y axes is greater thanside regions flanking the central region. Within the side regions, theinter-sensor distance 606 may be lesser than that within the centralregion where the optical sensors 120(7) are sparsely populated.

For illustrative purposes, an item outline 608 of an item 104 and anoutline of the hand 508 are depicted in the top view 602. The itemoutline 608 and corresponding shadow 132 that includes the footprint orshadow of the item 104 and the hand 508 are discussed in more detailbelow.

A controller 610 may be coupled to the optical sensors 120(7) of theoptical sensor array 120(13). The controller 610 may comprise amicrocontroller or other device configured to read out or otherwiseacquire information from the optical sensors 120(7). The controller 610may be configured to use the input from the optical sensors 120(7) togenerate the image data 422. In some implementations, the controller 610may provide the image data 422, such as a bitmap to another device suchas the server 204.

The side view 604 depicts additional components of the optical sensorarray 120(13). In some implementations, the optical sensors 120(7) maybe optically coupled to one or more optical element 612 devices. Theoptical elements 612 may comprise optical waveguides, optical fibers,mirrors, lenses, or other devices configured to direct, focus, control,or distribute at least a portion of incident light 506 to one or more ofthe optical sensors 120(7). The optical elements 612 may be arranged ina two-dimensional arrangement, while the optical sensors 120(7) may beotherwise arranged. For example, in one implementation, the opticalsensors 120(7) may be located along an edge of the optical sensor array120(13), and the optical elements 612 may comprise optical fibersmounted and configured as an array to gather the light 506 and directthe light 506 to the optical sensors 120(7).

In some implementations, a structure 614 may provide physical supportfor an item 104, may protect the optical sensor array 120(13) fromdamage, and so forth. The structure 614 may comprise a materialtransmissive to the wavelengths of light 506 that are detectable by theoptical sensors 120(7). For example, the structure 614 may compriseglass or plastic that is transparent or translucent. In someimplementations, the structure 614 may comprise a mesh or a materialwith holes through which light 506 may pass.

In the implementation depicted here, the items 104 rest upon thestructure 614. In other implementations, the items 104 may be supportedor suspended from above the structure 614, such as from support pegs orbars. The shadow 132 may comprise the shadow 132 cast by the hangingitems 104. For example, the items 104 may be hanging from a peg or ahook.

In the implementation depicted here, the optical sensor array 120(13) islocated below the item 104. The optical sensors 120(7) detect light 506from above the structure 614, such as passing through the shelf. Inother implementations, the optical sensor array 120(13) may be locatedin other positions relative to the item 104, such as above or behind.For example, the light source 504 and the optical sensor array 120(13)depicted in FIG. 6 may be transposed, such that the light 506 beneaththe structure 614 is emitted and directed upward toward the opticalsensor array 120(13). The shadow 132 may then be cast by the objectsbetween the light source 504 that is below onto the optical sensor array120(13) that is above. In another example, the optical sensor array120(13) may be arranged vertically, such as to the rear or one side ofthe partitioned area 134, to gather data about height of items 104.

One or more weight sensors 120(6) are configured to determine the weightof the load on the structure 614. For example, the weight sensors 120(6)may provide a physical coupling between the structure 614 and anotherportion of the inventory location 114, such as a support rib or frame.The weight sensors 120(6) may be arranged at various positions, such asproximate to the four corners of the structure 614.

Also depicted are cameras 120(1) configured to generate non-weight data128 such as image data 422. The FOV 130 of the cameras 120(1) areconfigured to include at least a portion of the inventory location 114.For example, the cameras 120(1) may be mounted above the shelf 502 andconfigured with the FOV 130 looking down on to the shelf 502.

FIG. 7 illustrates an overhead view 700 of partitioned areas 134 at theinventory location 114, the shadows 132 of objects on the partitionedarea 134, and weight distribution of the inventory location 114,according to some implementations.

As described above, a single inventory location 114 may be used to stowor hold different items 104. The inventory location 114 may bepartitioned into a plurality of partitioned areas 134. The inventorylocation 114 in this illustration is partitioned into a firstpartitioned area 134(1), a second partitioned area 134(2), and a thirdpartitioned area 134(3). A buffer zone 702 may be provided to improvedistinction between the partitioned areas 134.

As described above, partition data 330 may designate a particularportion or area of the inventory location 114 as being associated with aparticular partitioned area 134. For example, the partition data 330 maydescribe the size and the shape of the partitioned areas 134, theposition of the partitioned areas 134 with respect to the inventorylocation 114, and so forth.

In some implementations, the inventory location 114 may be serviced byone or more optical sensor arrays 120(13). During operation, the opticalsensor array 120(13) produces image data 422. The inventory managementmodule 324 may process the image data 422 to determine the presence ofone or more shadows 132. As the user 116 reaches for an item 104 stowedat the inventory location 114 in one of the partitioned areas 134, theirhand 508 casts an additional shadow 132 onto the optical sensor array120(13). For example, as depicted here the hand 508 of the user 116 isreaching to grasp one of the items 104 stowed at the first partitionedarea 134(1). As a result, a shadow 132(1) comprising the rectangularshadow 132 cast by the items 104 and the hand 508 is formed. Based onthis change in the shadow 132(1), interaction data 342 may be generatedthat indicates an interaction is occurring at the partitioned area134(1). Using the partition data 330, the partitioned area 134(1) may beassociated with the item identifier 402 of the item 104 stowed therein.

During some interactions, the shadow 132 before and after an interactionmay not change. For example, as the user 116 removes an item 104 from atop of a stack, such as an uppermost can of pet food, the shadow 132cast by the remaining cans sitting on the shelf remains unchanged.However, the processing module 328 may be configured to detect theshadow 132 cast by the hand 508, the item 104 as it is removed, and soforth.

In other interactions, the removal of an item 104 may provide additionalinformation to the processing module 328 that may be used to determineoccurrence of an interaction. For example, first image data 422(1) mayshow a rectangle with an area of 12 pixels while second image data422(2) acquired at a later time may indicate that the rectangle is nowsmaller with an area of 8 pixels. As a result, the interaction data 342may indicate an interaction type 454 of a “pick”. Likewise, an increasein the area of the shadow 132 may indicate an interaction type 454 of a“place”, as a newly placed item 104 blocks at least some of the light506.

In another implementation, the optical sensor array 120(13) may bemounted to gather data about a shadow 132 of a stack of items 104. Forexample, the optical sensor array 120(13) may be located on a side wallor partition separating the partitioned area 134(1) from partitionedarea 134(2) while the corresponding light source 504 may be arranged onthe opposite side of the partitioned area 134.

Similar techniques may be used to process other image data 422, such asobtained from one or more cameras 120(1) having a FOV 130 that includesat least a portion of the inventory location 114. For example, atransformation matrix may specify a correspondence between the locationof the pixels 424 in the image data 422 and the position of thepartitioned area 134 on the inventory location 114. As a result, thepresence of motion of an object in the image, such as a hand 508, may bedetermined and associated with a particular inventory location 114,partitioned area 134, or other location.

In some implementations, the processing module 328 may calculatequantity data 412 for a particular partitioned area 134. In oneimplementation, the calculation may use information based on the imagedata 422. For example, the area of a shadow 132 in the image data 422obtained from an optical sensor array 120(13) while in a stable state(such as when no hand 508 is present) may be divided by previouslystored geometry data 406 such as the area of the shadow 132 of anindividual item 104, and rounding the resulting value to a nearestinteger value. The integer value may then be used as the quantity data412.

In other implementations, the processing module 328 may calculatequantity data 412 for a particular partitioned area 134 using the weightdata 126. This calculation is described below in more detail.

A plurality of optical sensor arrays 120(13) may be used in conjunctionwith the same inventory location 114, partitioned area 134, and soforth. For example, two or more optical sensor arrays 120(13) mountedperpendicular to one another may be used to generate two sets ofnon-weight data 128 about the shadow of items 104. Based at least inpart on the shadow 132 as expressed by the non-weight data 128, a volumeoccupied by objects such as the items 104, the hand 508, and so forth,may be determined. The non-weight data 128 may also be used to generateinteraction data 342, to determine quantity data 412, and so forth.

As described above, the inventory location 114 may have one or moreweight sensors 120(6) to generate weight data 126 about a load. Forexample, as depicted here, weight sensors 120(6)(1) through 120(6)(4)are arranged at each of the four corners of an inventory location 114comprising a shelf 502. The weight sensors 120(6)(1) and 120(6)(3) areon a left side of the inventory location 114, while weight sensors120(6)(2) and 120(6)(4) are on a right side. In other implementations,the weight sensors 120(6) may be placed at other locations on orrelative to the inventory location 114. The load may include a portionof the shelf 502, other structures such as partitions, as well as theitems 104.

Each item 104 has inherent physical properties such as a weight,individual center-of-mass, height, width, depth, shape, and so forth. Agroup or collection of items 104 that are supported by or part of acommon structure have a combined weight distribution across a pluralityof weight sensors 120(6). An illustration of weight distribution data440 is depicted in a table that further indicates the side of theinventory location 114 that the weight sensor 120(6) is located on.

Individual objects have their own inherent center-of-mass (COM). Groupsof item 104, such as the entire inventory location 114 and the objectsstowed therein also have a COM. Depicted in this figure is indicia of acenter-of-mass (COM) 704 for the entire inventory location 114 includingthe items 104 stowed thereby, hardware on the shelf, and so forth. Inthis illustration, the COM 704 is located within the second partitionedarea 134(2). As illustrated with regard to FIG. 8, a change in thequantity or the arrangement of the items 104 may result in a change inweight distribution and the COM 704.

The COM 704 may be expressed in terms of coordinates with respect to anorigin. In some implementations, the COM 704 may be determined along asingle dimension, such as the width of the inventory location 114represented by the X axis in this figure. In this implementation, thevalues from the weight data 126 obtained from the weight sensors 120(6)located on the left side may be summed together to provide a single“left” weight data 126, while the values from the weight data 126obtained from the weight sensors 120(6) located on the right side may besummed together to provide a single “right” weight data 126. The COM 704for the inventory location 114 may thus be determined using the “left”weight data 126 and the “right” weight data 126, with the position ofthe COM 704 expressed as a linear measurement.

Also depicted is a location of weight (LWC) change 706. The LWC change706 in this illustration corresponds to the position, with respect tothe inventory location 114, of the COM 704 of the particular item 104that the hand 508 is removing. The LWC change 706 may be determined asdescribed above with regard to FIG. 4, in particular Equation 2.

The weight data 126 may be used to generate interaction data 342 inconjunction with the non-weight data 128. For example, the probabilitiesof different sets of hypotheses, some based on the weight data 126 andothers on the non-weight data 128, may be assessed. Continuing theexample, the hypothesis with the highest probability of being true maybe deemed the solution.

FIG. 8 illustrates a front view 800 of an inventory location 114 beforeand after removal of an item 104 from the inventory location 114,according to some implementations.

In this illustration, a front view is provided for three differenttimes, time=1 before an interaction, time=2 during the interaction, andtime=3 after the interaction. An origin 802 is designated at theleft-most edge of the inventory location 114. At the left and rightedges of the inventory location 114 are weight sensors 120(6).

Depicted here are the three partitioned areas 134(1), 134(2), and 134(3)in which items 104(1), 104(2), and 104(3), respectively are arranged inthe lanes. Distances from the origin 802 to the center of each of theitems 104 are indicated. For example, distance D1 indicates a distancefrom the origin 802 to the item 104(1), distance D2 indicates a distancefrom the origin 802 to the item 104(2), and distance D3 indicates adistance from the origin 802 to the item 104(3).

Based on the item data 136 indicative of the quantity of each item 104,the total weight of items 104 in each of the partitioned areas 134 maybe calculated. Using the total weight at each partitioned area 134 andthe distance data to the respective items 104, a distance to the firstCOM 704(1) (D-COM1) may be calculated.

At time=1, first weight data 126(1) is obtained from the weight sensors120(6) and used to determine D-COM1. A first weight distribution data440(1) may be generated from the first weight data 126(1).

At time=2, a quantity of 2 of item 104(1) has been removed from thepartitioned area 134(1), such as resulting from a pick by the user 116.

At time=3, after the interaction has completed, second weight data126(2) is obtained from the weight sensors 120(6) and used to determinedistance to the second COM 704(2) (D-COM2). A second weight distributiondata 440(2) may be generated from the second weight data 126(2).

Location of weight change (LWC) data 442 may be generated. For example,a difference between the first weight distribution data 440(1) and thesecond weight distribution data 440(2) may be used to determine the LWC706 as depicted here. Continuing the example, the difference between theweight distribution data may be used as input to Equation 2 describedabove.

A change in COM 804 may be determined by subtracting D-COM2 from D-COM1,or vice versa. The direction of the change in position of the COM 704along the inventory location 114 relative to the origin 802 may beindicated by the sign of the difference. For example, a change in COM804 having a positive sign may be indicative of a shift in the COM 704to the left, while a negative sign may be indicative of a shift in theCOM 704 to the right. The weight characteristic data 436 may include oneor more of the position of the COM 704 (such as the value of thedistance to the COM 704), change in COM 804 (both magnitude anddirection), and so forth. Similarly, a change in weight distribution maybe determined by subtracting the second weight distribution data 440(2)from the first weight distribution data 440(1).

The weight distribution data 440, location of the COM 704 relative tothe inventory location 114, the LWC 706, the change in COM 804, or otherweight characteristic data 436 may be used by the processing module 328to determine interaction data 342 by selecting or discarding varioushypotheses. For example, the change in weight of the inventory location114 and the LWC 706 may be indicative of the removal of two items 104(1)from the partitioned area 134(1). The LWC 706 may be used to associatean interaction with a particular partitioned area 134 at the inventorylocation 114. The particular partitioned area 134 may be configured byhypotheses data based on non-weight data 128, such as image data 422indicative of which partitioned area 134 experienced the occurrence ofan event, such as the presence of the user's 116 hand 508.

As described above, various predicted interactions may be expressed ashypotheses data 340. The hypotheses data 340 may comprise differentcombinations of quantities of different items 104, their respectiveplacement within partitioned areas 134, and so forth. By assessing theprobabilities that the hypotheses are true, the interaction data 342 maybe determined by the inventory management system 122.

Illustrative Processes

FIG. 9 depicts a flow diagram 900 of a process for determininginteraction data 342 using different sets of hypotheses, according tosome implementations. In some situations, the hypotheses data 340 thatis based on the sensor data 124 from a particular sensor may not exhibita hypothesis with a confidence value sufficient to be designated as asolution. As a result, the determination of the solution may involvehypotheses data 340 for hypotheses based on one or more of differentsets of sensor data 124 or sensor data 124 processed in different ways.By fusing the sensor data 124 from different types of sensors 120,interaction data 342 that is true to the measured interaction may bedetermined.

The process may be implemented, at least in part, by the inventorymanagement module 324. In different implementations, one or more of theblocks described below may be omitted, the sequence of the process usingthese blocks may vary from that depicted, and so forth. The sensors 120may provide information about the same portion of the facility 102, suchas a particular inventory location 114.

An image processing module 902 accesses image data 422. For example, theimage data 422 may comprise one or more images acquired from one or morecameras 120(1). The one or more cameras 120(1) may have a FOV 130 thatcovers at least a portion of an inventory location 114. For example,camera 120(1)(1) may have a FOV 130 that includes partitioned area134(1), while camera 120(1)(2) may have a FOV that includes anotherpartitioned area 134(1)(2), and so forth.

The image processing module 902 may use one or more image processingtechniques to determine information based on the image data 422. Thisinformation may include motion data, differential data, data indicativeof an item identifier 402, and so forth.

The image processing module 902 may determine motion data indicative ofmotion, at a partitioned area 134, of one or more objects between agroup of frames of the image data 422. For example, motion detection mayinvolve processing a group of frames by subtracting one consecutiveimage in the group from another, such as using the “cvSub( )” functionof OpenCV. A count of the number of pixels 424 in the resultingdifferential images with non-zero values may be determined. If the countexceeds a threshold value, motion may be determined to have occurredbetween the two frames. If a sequence of counts exceeds a threshold, thegroup of frames may be designated as indicative of motion. In otherimplementations, other techniques may be used.

In another example, the motion detection may involve determining agradient difference between pixels in consecutive frames of the imagedata 422. Motion in the image data 422 may be determined based on thegradient difference exceeding a threshold value.

The image processing module 902 may determine differential dataindicative of items 104 that have been removed or placed at one or moreof the partitioned areas 134 at the inventory location. For example, theimage processing module 902 may determine if there is a difference inimages at a start before and at an end after the motion. A differencebetween the before and after images may be indicative of a changeoccurring at the partitioned area 134. The change may be due to anaddition or removal of an item. In some implementations, the imageprocessing module 902 may detect indicia or markings on the surface ofthe partitioned area 134. For example, tracking marks, bar codes, aruler, and so forth may be printed on the surface of the inventorylocation. By comparing the indicia in the start and end frames, adetermination may be made as to whether items 104 have been added orremoved from the partitioned area 134.

Based on the information about a particular partitioned area 134, theitem data 136 may be used to determine what items 104 are stowed at thatparticular partitioned area 134. For example, the partitioned area134(1) may be associated with item identifier 402 of “12345”.

Using the image data 422 and the item data 136, the image processingmodule 902 may determine a first set of hypotheses 904. The first set ofhypotheses 904 may be provided to a first fuser module 906, describedbelow. The first set of hypotheses 904 may include one or more itemidentifiers 402 with associated probability values for each hypothesis.The probability values may indicate a probability that a hypothesis istrue with respect to the item identifier 402 that was involved in theinteraction at the inventory location 114. For example, one of thehypotheses in the first set of hypotheses 904 may indicate that a singletype of item SKU #12345 was involved in the interaction, with aprobability of this being true of 0.95.

In some implementations, generation of the first set of hypotheses 904may be based upon occurrence of a triggering event. The triggering eventmay be the determination of the motion, determination of a change at thepartitioned area 134, and so forth, as described above. For example,upon a determination of occurrence of motion, the first set ofhypotheses 904 may be determined.

A weight processing module 908 is configured to access the weight data126 and the item data 136. The weight processing module 908 may beconfigured to determine a second set of hypotheses 910. The second setof hypotheses 910 may include one or more of: item identifiers 402,predicted quantities for each of the one or more items 104, orprobability values indicative of a probability that the hypotheses aretrue with respect to the interaction at the inventory location.

In some implementations, the second set of hypotheses 910 may alsoinclude predicted weight characteristic data 436. For example, thesecond set of hypotheses 910 may include a predicted weight distributionacross a plurality of weight sensors 120(6) for the predicted quantityof the predicted item 104.

The weight processing module 908 may use the weight data 126 todetermine weight characteristic data 436, and the second set ofhypotheses 910 therefrom. As described above, various permutations ofdifferent quantities of those items 104 may be used to determine thesecond set of hypotheses 910. In some implementations, constraint datamay be used to constrain the hypotheses in the second set of hypotheses910. For example, measured weight change data 438 may be used to limitthe second set of hypotheses 910 to those having predicted weightchanges within a threshold range.

In some implementations, generation of the second set of hypotheses 910may be based upon occurrence of a triggering event. For example, theweight processing module 908 may use a cumulative sum (CUSUM) techniqueto determine that an event is taking place at the weight sensors 120(6).Use of the CUSUM technique may help determine that a change in weight isnot the result of noise or other spurious occurrences, but due to ameasured change in the load upon the weight sensors 120(6). Oneimplementation of the CUSUM technique is discussed below in more detailwith regard to FIG. 12.

The first set of hypotheses 904 and the second set of hypotheses 910 maybe provided to a first fuser module 906. The first fuser module 906 maybe configured to produce a third set of hypotheses 912 using at least aportion of the first set of hypotheses 904 and at least a portion of thesecond set of hypotheses 910. In some implementations, sets ofhypotheses may be combined using Bayes' rule. For example, Bayes' rulemay be used to aggregate the first set of hypotheses 904 and the secondset of hypotheses 910 into the third set of hypotheses 912 by computinga joint probability of the hypotheses representative of concurrentevents. In other implementations, other techniques may be used to join,combine, merge, or otherwise utilize the information in both of the setsof hypotheses to generate the third set of hypotheses 912.

In some situations, one or more of the sensors 120 used to determinesensor data 124 may be unavailable, may produce incomplete, low quality,or unusable sensor data 124, and so forth. For example, one or more ofthe weight sensors 120(6) may be producing erroneous weight data 126,one or more of the cameras 120(1) may be inoperative and not beproviding image data 422, and so forth.

If one set of hypotheses is empty, incorrect results may be returned ifthe empty set is combined with a non-empty set of hypotheses. Forexample, if image data 422 is unavailable, the first set of hypotheses904 may have zero probability values. When this zero value is multipliedwith a probability in the corresponding second set of hypotheses 910,each resulting hypothesis in the third set of hypotheses 912 would havea probability of zero. Various techniques may be used to address thesituation where some of the sensor data 124 used by the processingmodule 328 is unavailable.

One technique involves the use of predetermined placeholder data whichmay be used in place of absent or erroneous sensor data 124. Forexample, when the image data 422 is unavailable, the first fuser module906 may use a predetermined factor or function in place of the missingimage data 422. The probabilities of the hypotheses in the second set ofhypotheses 910 may be modified using this predetermined factor orfunction, and the results may be provided as the third set of hypotheses912. Continuing the example, the predetermined factor may have a valueof “0.5” and the probabilities for each of the second set of hypotheses910 may be multiplied by the same value of “0.5”. Additional thresholdsmay be used to determine the factor to apply. For example, where thepredicted weight in the second set of hypotheses 910 is above athreshold, the factor of “0.5” may be used. Continuing the example, afactor of “0” may be used when the predicted weight is below thethreshold.

Another technique involves assessing the complexity of the hypothesesprovided to the fuser module. For example, a hypothesis may be deemedcomplex if it involves a pick and a place. In the event sensor data 124is unavailable, the fuser module may disqualify hypotheses frominclusion in the output set of hypotheses based on the complexity. Forexample, the image data 422 may be unavailable to the image processingmodule 902 or may be of inadequate quality, resulting in an empty orincomplete first set of hypotheses 904. The second set of hypotheses 910based on the weight data 126 may be processed to determine whichhypotheses in that second set are complex or non-complex. Thenon-complex hypotheses of the second set of hypotheses 910 may beincluded in the third set of hypotheses 912 while the complex hypothesesmay be omitted.

By using these techniques, the probabilities in the set of hypothesesdetermined by the fuser module are decreased. As the sensor data 124becomes unavailable or is of poor quality, the output from the fusermodule may be degraded, but operation of the system overall maycontinue. Likewise, as sensor data 124 becomes available or improves inquality, the output from the fuser module improves.

The third set of hypotheses 912 may include one or more item identifiers402 and quantities with probability values indicative of a probabilitythe hypothesis is true with respect to the interaction at the inventorylocation 114. By combining the first set of hypotheses 904 and thesecond set of hypotheses 910, the third set of hypotheses 912 providesfor disambiguation between competing hypotheses. For example, the secondset of hypotheses 910 may contain several hypotheses with similarprobabilities. By combining with the first set of hypotheses 904, thesituation may be disambiguated.

The third set of hypotheses 912 may be sorted and ranked according tothe probability of each hypothesis. For example, the hypotheses may besorted in descending order of probability. The ranking may result indesignation of a first hypothesis having a first highest probability anda second hypothesis having a second highest probability.

The confidence value for the first hypothesis may be determined. Forexample, the confidence value may comprise a difference between thefirst highest probability and the second highest probability.

At 914, the confidence value of the first hypothesis of the third set ofhypotheses 912 may be compared to a confidence threshold value. When theconfidence value meets or exceeds the confidence threshold value, thefirst hypothesis may be designated as a “high confidence” hypothesis.The process may proceed to designate the first hypothesis as a solution,which then may be used to determine the interaction data 342.

In comparison, when at 914 the confidence value is below the confidencethreshold value, the first hypothesis may be designated as a “lowconfidence” hypothesis. In this situation, the third set of hypotheses912 that includes a low confidence first hypothesis may be provided to asecond fuser module 916.

Returning to the image processing module 902, the image processingmodule 902 may produce as output start and end frames 918. The start andend frames 918 may comprise those frames that occur before and after anevent, such as detection of motion within the group of images. Forexample, the before frame may be determined as a frame occurring priorto the detection of motion while the after frame may be determined as aframe occurring following the conclusion of the motion that wasdetected. The start and end frames 918 may be provided to a count module920.

The count module 920 may be configured to use one or more machine visioncounting techniques to determine a change in count of the items 104 atthe partitioned area 134. For example, a machine vision countingtechnique may be configured to identify a top portion of the items 104as acquired from a camera 120(1) that is looking down on the shelfholding the items 104. This determination may be based on item data 136,such as previously acquired images of an exemplary item 104. Each of thetops of the type of item 104 appearing in a frame may be identified, anda count made.

In another implementation, indicia on the inventory location 114 may beobserved. For example, a ruler or other markings may be printed on thesurface of the inventory location 114. Instead of, or in addition to,identifying the items, a change in the observed indicia may bedetermined and used to count. Continuing the example, the frame mayinclude a portion of a ruler printed on the shelf. The length of theruler visible in a frame may be determined. Given the length visible (ora change therein) and given information about a dimension of the type ofitem 104 (such as the depth), a count may be calculated.

A difference in the counts between the first image and the second imagemay be determined. For example, the first image may result in a count of10 cans of pet food while the second image may result in a count of 8cans of pet food. A hypothesis that 2 cans were removed may bedetermined, along with a probability that this hypothesis is accurate.The count module 920 may determine a fourth set of hypotheses 922. Thefourth set of hypotheses 922 may include one or more quantities withprobability values indicative of a probability that the hypothesis istrue with respect to the interaction at the inventory location 114. Forexample, the fourth set of hypotheses 922 may indicate “quantity 2,probability 0.95”.

In other implementations, the count module 920 may be configured to useother non-weight data 128 to determine the fourth set of hypotheses 922.For example, the count module 920 may receive data from an auto-facingunit or other sensor 120 at the inventory location 114. The auto-facingunit may comprise a position sensor configured to provide dataindicative of displacement of a pusher. As an item 104 is removed fromthe auto-facing unit, the pusher moves, such as under the influence of aspring, and pushes the remaining items 104 in the auto-facing unit tothe front of the inventory location 114. By using data from the positionsensor, and given item data 136 such as a depth of an individual item104, the count module 920 may determine a count based on a change inposition data. For example, if each item 104 is 1 inch deep, and theposition data indicates a change of 3 inches, the quantity may havechanged by 3 items 104.

The second fuser module 916 uses the third set of hypotheses 912 and thefourth set of hypotheses 922 to form a fifth set of hypotheses 924. Thefifth set of hypotheses 924 may include one or more item identifiers402, the predicted quantity for each of the one or more item identifiers402, and probability values indicative of a probability that the itemidentifier 402 and predicted quantity is true with respect to theinteraction at the inventory location 114. Various techniques may beused to join, combine, merge, or otherwise utilize the information inboth of the sets of hypotheses to generate the fifth set of hypotheses924. For example, Bayes' rule may be used to produce joinedprobabilities of hypotheses from each set.

The fifth set of hypotheses 924 may be sorted in descending order of theprobability for each hypothesis, and ranked. As described above, aconfidence value for the first ranked hypothesis of the fifth set ofhypotheses 924 may be determined. At 926, the first ranked hypothesis ofthe fifth set of hypotheses 924 may be compared to the confidencethreshold value. When the confidence value of the first rankedhypothesis of the fifth set of hypotheses 924 meets or exceeds theconfidence threshold value, the first hypothesis may be designated as a“high confidence” hypothesis. The process may proceed to designate thefirst hypothesis as a solution, which then may be used to determine theinteraction data 342.

In some implementations, at 926 a different confidence threshold valuemay be used, as compared to the confidence threshold value at 914. Forexample, the confidence threshold value used at 914 may be 0.6 while theconfidence threshold value used at 926 may be 0.5.

In comparison, when at 926 the confidence value is below the confidencethreshold value, the first hypothesis may be designated as a “lowconfidence” hypothesis. In this situation, the fifth set of hypotheses924 may be provided to a manual assessment module 928. For example, themanual assessment module 928 may present at least a portion of the imagedata 422 to a human operator who may then select one or more of thehypotheses from the fifth set of hypotheses 924 as the solution or whomay determine a new hypothesis.

In some implementations, temporary or interim interaction data may bedetermined. For example, when block 914 or 926 determines that theconfidence value or another metric is below a threshold, the process mayproceed contemporaneously to block interaction data 342 and also toblocks 916 or 928, respectively. This allows the process to proceedusing the interim interaction data 342, reducing latency, while longerrunning processes such as the use of additional fuser modules are takingplace. At a later time, when further information is available, such asthe fifth set of hypotheses 924 or the manual input, the interiminteraction data 342 may be replaced with final interaction data 342.

Returning to 914, in one implementation, the determination of thehypothesis with a low confidence value may be used to trigger thegeneration of the start and end frames 918, and subsequent processing bythe count module 920 to determine the fourth set of hypotheses 922. Thismay be done to reduce computational overhead associated with theperformance of the count module 920.

In another implementation, the image processing module 902 may determinethe start and end frames 918 and provide them for processing to thecount module 920 in parallel to the operations described above withrespect blocks 902 through 914. In this implementation, the second fusermodule 916 may experience a delay while waiting for the fourth set ofhypotheses 922 to be provided by the count module 920.

The sets of hypotheses described herein, such as the third set ofhypotheses 912, may be configured to include a default hypothesisindicative of a pick and place of an item 104 to and from the inventorylocation 114 but with a net change of zero. For example, this hypothesismay indicate that item 104(13) has a pick and place quantity of 0 atpartitioned area 134(3). By including the default hypothesis in the setof hypotheses, spurious or no change events, such as inspection orbumping of an item 104, may be properly recognized.

While a first fuser module 906 and a second fuser module 916 aredepicted, additional fuser modules (not shown) may also be utilized. Forexample, a third fuser module, fourth fuser module, and so forth, may beused by the system 100.

In some implementations, one or more of the fuser modules may accessother data. For example, user data associated with a particular user 116or category of user may be accessed. The user data may include one ormore of pick history, place history, shopping list data, personalpreference data, language preferences, food allergies, and so forth. Thefuser module may utilize this data to determine probabilities associatedwith hypotheses, to constrain hypotheses, and so forth. For example, thefirst fuser module 906 may access user data indicative of the user 116having a severe allergy to peanuts. The first fuser module 906 mayassign a lower priority to hypotheses involving items 104 containingpeanuts, under the assumption that the user 116 may avoid these items104.

The fuser modules may also be configured to selectively indicate thatthe outputs are of low confidence. This situation may arise whenhypotheses to be fused do not have explicit agreement with one another.For example, the first set of hypotheses 904 may have hypothesesassociated with first item identifier 402(1), while the second set ofhypotheses 910 has hypotheses associated with second item identifier402(2). If the first item identifier 402(1) does not appear in thesecond set of hypotheses 910, or that second item identifier 402(2) doesnot appear in the first set of hypotheses 904, the resulting output inthe third set of hypotheses 912 may be designated as having lowconfidence.

FIG. 10 depicts a flow diagram 1000 of a process for generatinginteraction data 342 indicative of an interaction such as a pick orplace of an item 104, according to some implementations.

The process may be implemented at least in part by the inventorymanagement module 324. In different implementations, one or more of theblocks described below may be omitted, the sequence of the process usingthese blocks may vary from that depicted, and so forth. The sensors 120may provide information about the same portion of the facility 102, suchas a particular inventory location.

Block 1002 accesses weight data 126 acquired by a plurality of weightsensors 120(6) at an inventory location 114. For example, the server 204may receive the weight data 126.

Block 1004 accesses non-weight data 128 that is associated with theinventory location 114. For example, the non-weight data 128 maycomprise the image data 422 acquired by one or more of the cameras120(1) that include a FOV 130 of at least a portion of the inventorylocation 114.

Block 1006 accesses item data 136. As described above, the item data 136may comprise previously stored information such as item identifiers 402associated with particular partitioned areas 134 at the inventorylocation 114.

Block 1008 determines the weight data 126 and the non-weight data 128such as the image data 422 occur contemporaneously with one another intime and space. For example, the data may be deemed to occurcontemporaneously when timestamps associated with each are within awindow or threshold range of time of one another and when they share acommon portion of the facility 102, such as a particular inventorylocation 114.

Block 1010 generates, from the non-weight data 128, motion data 446. Forexample, the image data 422 may be processed to determine motion withina group of frames. For example, the sequence of frames recorded by thecamera 120(1) may be processed to determine motion between theconsecutive frames. In another example, data from one or more proximitysensors 120(14) may be used to determine motion data 446.

Block 1012 determines, from a start frame acquired before the occurrenceof motion and an end frame acquired after the occurrence of motion,differential data 434 indicative of a change in appearance of theinventory location 114. For example, the start frame may be determinedas occurring before the motion determined by block 1010 while the endthe frame may be determined as occurring after the motion. In oneimplementation, the differential data 434 may be determined bysubtracting the end image from the start image or vice versa. A changein appearance may be determined when the number of pixels 424 in thedifferential data 434 exhibiting a difference exceeds a threshold value.Other implementations of the processing techniques may be used, such ascontour detection, comparisons of changes in contour data between thestart and end images, and so forth.

Block 1014 determines, based on the non-weight data 128 such as theimage data 422, a first set of hypotheses 904 including one or more itemidentifiers 402 with probability values indicative of a probability thatthe item identifier 402 is true with respect to an interaction at theinventory location 114.

Block 1016 determines, based on to the weight data 126, a second set ofhypotheses 910. The second set of hypotheses 910 may include one or moreitem identifiers 402, a predicted quantity for each of the one or moreitem identifiers 402, and probability values indicative of a probabilitythat the hypothesis is true with respect to the interaction at theinventory location 114.

Block 1018 determines, based on the first set of hypotheses 904 and thesecond set of hypotheses 910, a third set of hypotheses 912. Forexample, the first fuser module 906 may determine the third set ofhypotheses 912 that includes one or more item identifiers 402 andquantities with probability values indicative of a probability thehypothesis is true with respect to the interaction at the inventorylocation 114. In some implementations the sets of hypotheses may becombined using Bayes' rule.

Block 1020 determines a first confidence value for the first hypothesisbased on a difference between the first highest probability and thesecond highest probability of the third set of hypotheses 912. Forexample, the hypotheses in the third set of hypotheses 912 may be sortedby descending order of probability and ranked to designate a firsthypothesis having a first highest probability and a second hypothesishaving a second highest probability.

In other implementations, instead of or in addition to the confidencevalue, other metrics may be determined. For example, the value of theprobability of the respective hypotheses may be used instead of theconfidence value.

Block 1022 determines the confidence value of the first hypothesis isbelow a confidence threshold value. For example, the probabilities forthe first hypothesis and the second hypothesis may be relatively closeto one another, resulting in a potential ambiguity between hypothesesand the measured interaction.

In some implementations, instead of or in addition to a determinationusing the confidence value, other metrics or input may be used todetermine how accurately the first hypothesis corresponds to an actualsituation. For example, the probability of the first hypothesis asdescribed above may be determined to be below a threshold value.

Block 1024 determines, responsive to one or more of the motion data orthe differential data 434, and based on the start frame and the endframe 918, a fourth set of hypotheses 922. The fourth set of hypotheses922 may include one or more quantities with probability valuesindicative of a probability that the hypothesis is true with respect tothe interaction at the inventory location 114. For example, the countmodule 920 may determine the fourth set of hypotheses 922.

Block 1026 determines, based on the third set of hypotheses 912 and thefourth set of hypotheses 922, a fifth set of hypotheses 924. The fifthset of hypotheses 924 may include one or more item identifiers 402, thepredicted quantity for each of the one or more item identifiers 402, andprobability values indicative of a probability that the item identifier402 and predicted quantity is true with respect to the interaction atthe inventory location 114.

Block 1028 determines a second confidence value for a third hypothesisof the fifth set of hypotheses 924. The hypotheses of the fifth set ofhypotheses 924 may be sorted in descending order of probability and thetop two hypotheses may be designated. For example, the third hypothesismay be the first ranked hypothesis of the fifth set of hypotheses 924and have the first highest probability for that set. In comparison, afourth hypothesis may be a second ranked hypothesis of the fifth set ofhypotheses 924 and have a second highest probability for that set.

Block 1030 determines the second confidence value is at or above aconfidence threshold value. In some implementations, the confidencethreshold value used to compare the second confidence value may differfrom the confidence threshold value used to compare the first confidencevalue.

Block 1032 designates the third hypothesis as a solution. Continuing theexample, given the second confidence value exceeding the confidencethreshold value, the third hypothesis is deemed to have a probability oftruth that is sufficiently high.

Block 1034 determines, based on the solution, interaction data 342. Theinteraction data 342 may be indicative of one or more of a change inquantity of the one or more items 104 resulting from the interaction, anitem identifier 402 indicative of the one or more items 104 involved inthe interaction, data indicative of the partitioned area 134 at whichthe one or more of the items 104 were picked or placed, and so forth.

In some implementations, messages or other mechanisms may be used tocoordinate the different activities associated with processing thesensor data 124. This coordination may involve waiting for additionalsensor data 124 to complete acquisition. For example, the weight data126 may be determined in under a second, while the non-weight data 128such as the image data 422 may require several seconds for acquisition.

A first message may be received from the weight sensors 120(6)indicative of a change in state, such as from a stable to an unstablecondition, or vice versa. This message indicates that an event may betaking place that may provide data useful to the system. A secondmessage may be received from the camera 120(1) or other non-weightsensor indicative of a change. For example, the camera 120(1) may haveonboard motion detection processing and may provide a message indicativeof a change between two or more consecutive frames.

These messages may be used to alert one or more of the modules of theinventory management module 324 that additional data may be forthcoming.For example, the first fuser module 906, the second fuser module 916,and so forth, may be configured to wait to process data until they havereceived the sensor data 124 that is expected.

A third message may be received from the weight sensors 120(6)indicative of the change concluding. For example, the weight sensor120(6) may determine that no changes have taken place in the measuredweight for a threshold amount of time and may then determine the thirdmessage.

A fourth message may be received from the non-weight sensors 120 such asthe camera 120(1), indicative of the change concluding. For example, theonboard motion detection processing of the camera 120(1) may indicatethat no further changes have been observed between consecutive framesand determine the fourth message.

These messages, or data indicative of them, may be provided to therespective modules within the processing module 328 for further use. Forexample, one or more of the first fuser module 906 or the second fusermodule 916 may be configured to wait to determine their respective setsof hypotheses until they receive the third and fourth messagesindicating that changes measured by the respective sensors 120 haveconcluded. By providing for this additional information, the inventorymanagement module 324 may avoid inadvertently processing data before atransaction is concluded. Should the concluding message (such as thethird and fourth messages) never be received, the process may time out.After timing out, the fuser module may continue to process the sensordata 124 that is available. Should a time out occur during processing,the solution associated with that time out may be designated as a lowconfidence event.

In some implementations, the messages may be determined by one or moreother modules. For example, the image processing module 902, the weightprocessing module 908, and so forth, may determine the messages.

In some situations, messages may be subject to a timeout window. Forexample, if the third or fourth message indicating that changes areconcluded are not received within a threshold amount of time, theprocessing of the previously received sensor data 124 may proceed.

FIG. 11 depicts a flow diagram 1100 of another process for generatinginteraction data 342, according to some implementations. The process maybe implemented at least in part by the inventory management module 324.In different implementations, one or more of the blocks described belowmay be omitted, the sequence of the process using these blocks may varyfrom that depicted, and so forth. The sensors 120 may provideinformation about the same portion of the facility 102, such as aparticular inventory location 114.

Block 1102 determines, based on non-weight data 128, a first set ofhypotheses. For example, image data 422 acquired from a camera 120(1)viewing the inventory location 114 may be accessed. The image data 422may be processed by the image processing module 902 to determine thefirst set of hypotheses 904. The item data 136 indicative of placementof one or more items 104 at one or more partitioned areas 134 of theinventory location 114 may also be accessed and used by the imageprocessing module 902. In some implementations, each hypothesis in thisset may be indicative of a probability that a particular item identifier402 was involved in an interaction.

Block 1104 determines, based on the weight data 126, a second set ofhypotheses. For example, the weight data 126 from a plurality of weightsensors 120(6) at an inventory location 114 may be accessed. The itemdata 136 indicative of weight of one or more items 104 and theirplacement at one or more partitioned areas 134 of the inventory location114 may also be accessed and used by the weight processing module 908.The weight data 126 may be processed by the weight processing module 908to determine the second set of hypotheses 910. In some implementations,each hypothesis in this set may be indicative of a probability that aparticular quantity of a particular item 104 was involved in theinteraction.

In some implementations, the second set of hypotheses 910 may be basedon one or more constraints. For example, a block (not shown) determinesa measured change in weight at the inventory location 114. Another blockdetermines a measured weight distribution of the inventory location 114.One or more hypotheses may be determined, in which each hypothesiscomprises data indicative of: a predicted quantity of the particularitem 104, a predicted change in weight, and a predicted weightdistribution. A subset of the one or more hypotheses may be determinedas the second set of hypotheses 910 based at least in part on one ormore of the measured change in weight or the measured weightdistribution. For example, the second set of hypotheses 910 may berestricted to hypotheses that have a predicted weight change that iswithin a threshold value of the measured weight change.

In some implementations, the weight data 126 and the non-weight data 128that are used to determine sets of hypotheses may be selected such thatportions thereof are determined to have occurred contemporaneously intime and space with one another. For example, at least a portion of thesensor data 124 may have timestamps, sequence numbers, or other indiciathat may be used to specify an interval of time or commonality ofoccurrence. The contemporaneous occurrence may include simultaneousoccurrences or those within a threshold time of one another. Forexample, the weight data 126 and the non-weight data 128 havingtimestamps within a threshold window of 150 ms may be deemed to becontemporaneous with one another. To determine whether the data arespatially contemporaneous, physical configuration data such asindicating what sensors 120 gather information for what portions of thefacility 102 may be used.

Block 1106 determines a third set of hypotheses 912 based at least inpart on at least a portion of the first set of hypotheses 904 and atleast a portion of the second set of hypotheses 910. For example, Bayes'rule may be used to aggregate the first set of hypotheses 904 and thesecond set of hypotheses 910 by computing a joint probability of thehypotheses representative of concurrent events.

In some implementations, an occurrence of motion at the inventorylocation 114 may be determined. The detection of motion may be based onchanges between a plurality of images in the image data 422. Changesbetween successive images may be indicative of motion of an objectwithin the FOV 130 of the camera 120(1) that acquired the images. Adetermination of one or more of cessation of motion in the image data422 or that a maximum wait time has been reached may be made. Thegeneration of the third set of hypotheses 912 may be responsive to thecessation.

One or more additional hypotheses may be inserted to or injected intothe third set of hypotheses 912. For example, a default hypothesisindicative of a pick and place of an item 104 to and from the sameinventory location 114 with a zero net change of item quantity may beinserted. Insertion of this default hypothesis may improve overallsystem performance by avoiding incorrect determinations of hypothesesthat may result from interactions such as bumping or touching an item104. A probability that this default hypothesis is true may also becalculated and included in the ranking and analyses of the hypothesis asdescribed above.

Block 1108 determines a confidence value for one or more of thehypotheses in the third set of hypotheses 912. For example, theconfidence value may be calculated by subtracting a probability of asecond ranked hypothesis from a probability of a first ranked hypothesisin the third set of hypotheses 912.

Based on the confidence value, block 1110 determines a solutionindicative of a quantity of items 104 that changed at the inventorylocation 114. For example, when the confidence value of a hypothesismeets or exceeds confidence threshold values, that hypothesis may bedesignated as the solution.

As described above, in some situations the probabilities of varioushypotheses in the set of hypotheses may be insufficient to reach orexceed the confidence threshold value. In these situations, additionalhypotheses may be determined using sensor data 124 acquired from othersensors 120 or from sensor data 124 that has been processed in adifferent fashion. These additional hypotheses may be combined with oneor more previously determined sets of hypotheses to determine a highconfidence hypothesis that may be designated as a solution.

For example, a block (not shown) may determine, from the image data 422,data indicative of occurrence of motion at the inventory location 114 asrecorded by a group of frames. From this group of frames, a start framebefore the group of frames and an end frame after the group of framesmay be determined. The start and end frames 918 may then be used todetermine differential data 434 indicative of a change in appearance ofthe inventory location 114. The motion may indicate that some event istaking place at the partitioned area 134, while the differential data434 provides more information about whether that event involved addingor removing items 104.

Continuing the example, a fourth set of hypotheses 922 may be determinedusing the start and end frames 918. The fourth set of hypotheses 922 mayinclude one or more quantities with probability values. The probabilityvalues are indicative of a probability that the quantity in thehypothesis was one or more of: added to or removed from the inventorylocation 114 or a portion thereof, such as the partitioned area 134. Thefourth set of hypotheses 922 may be combined by the second fuser module916 with one or more previously determined sets of hypotheses.

In one implementation, the fourth set of hypotheses 922 may bedetermined by the count module 920. The count module 920 may beconfigured to use the start and end frames 918 to count the number ofitems 104 appearing in the images. For example, the count module 920 maycount a start number of item tops, sides, bottoms, or other featuresappearing in the start frame. The count module 920 may also count an endnumber of the features, such as item tops, appearing in the end frame.The count module 920 may determine a quantity of the items 104 that wasone or more of added to or removed from the partitioned area 134 bysubtracting the start number from the end number. The count module 920may use the determined quantity to determine a hypothesis. Thishypothesis may include a probability indicative of whether thehypothesis is true.

Block 1112 determines interaction data 342 based at least in part on thesolution. For example, the predicted values present in the hypothesisdesignated as a solution may be used as measured values indicative ofthe interaction.

Another block (not shown) may use the information in the interactiondata 342 to change item data 136. For example, the predicted valuespresent in the hypothesis may be used to change data indicative of aquantity on hand of the item 104 at the partitioned area 134 of theinventory location 114.

FIG. 12 depicts a flow diagram 1200 of a process of generating weightdata 126, according to some implementations. In some implementations,the process may be performed at least in part by weight processingmodule 908, or by a processor at the inventory location 114 and coupledto the weight sensor 120(6).

Block 1202 accesses the “raw” weight data 126 from the weight sensors120(6). In some implementations, the raw weight data 126 may have somepre-processing applied.

Block 1204 determines filtered sensor data from the raw weight data 126.For example, a filter module may process the raw weight data 126 usingan exponential moving average function.

Block 1206 accesses a window length indicative of an interval of time.For example, the window length may be specified by configuration data.In some implementations, the window length may be between at least 400milliseconds and at most 600 milliseconds. For example, the windowlength may be about 500 ms. In implementations where the weight data126, the filtered sensor data, or both are expressed in terms ofindividual samples, frames, or other discrete data elements, windowlength may be specified in terms of a number of those discrete dataelements. For example, window length may be specified as 15 samples.

Block 1208 accesses a portion of the filtered sensor data associatedwith a range of time values delineated by the window length. Continuingthe example above, the 15 most recent samples may be accessed.

Block 1210 determines a line based on the filtered sensor data occurringwithin the window length. For example, a slope module may fit a line tothe filtered sensor data using a linear regression function. Continuingthe example, the linear regression function may comprise a least squareregression function.

Block 1212 determines slope value data of the line. Continuing theexample, the slope module may calculate the slope of the line andprovide that output as the slope value data. As described above, theslope value data may be visualized as being indicative of a change inrise over a run of the line, or how “steep” the line is.

Block 1214 determines change data associated with the slope value data.For example, an alarm module may determine alarm data which is thenprocessed by a change detector module to determine the change data. Insome implementations, the alarm module may implement a cumulative sum(CUSUM) function. The CUSUM function is a sequential analysis techniqueoriginally developed by E. S. Page. The CUSUM function may be configuredto analyze changes to a plurality of slope values over time. Oneimplementation of the alarm function using a variant of the CUSUMfunction is described below with regard to Code Implementation 1.

CODE IMPLEMENTATION 1 ″″″Runs cusum on a scalar value and alarms ifthere are changes.″″″ from abc_alarm_algorithm import AlarmAlgorithmfrom abc_alarm_algorithm import AlarmEventType import abc metrics classScalarCusumAlgorithm(AlarmAlgorithm):  ″″″An implementation ofcumulative sum alarming.  Takes in a signal estimate and raw signal andmaintains positive and negative residuals. Alarms if the residuals crossa threshold.  NOTE: Does not maintain the signal estimate.  A note aboutthe H and V design parameters:  H:  If the cumulative sum of thepositive or negative residual (delta of the raw signal and signalestimate) exceeds this value, an alarm is raised and the cumulative sumsare reset. This is related to the noise of the system and what theoperator wants to be defined as a change.  A good value to start with is100. If too many changes are detected or the system is constantlyunstable, try doubling until a reasonable output is obtained.  To getmore changes, lower this value. To get fewer changes, raise this value. V:  At each new residual, this value is subtracted from the positiveand negative residual. This value is used to account for drift in thesystem that could cause false cusum alarms. A good value to start withis 5. If constantly unstable or seeing slow events cause changes tryraising this value. If not seeing any changes try lowering this value. ″″″  def_init_(self,    H,    V,    name=“ScalarCusumAlgorithm”,   unpack_method=None,    pack_method=None):  ″″″Initializes thealgorithm.  keyword parameters:  H - The H design parameter as a float. V - The V design parameter as a float.  name - The name to initializethe alarm algorithm with as a string. (Default ScalarCusumAlgorithm).Can be used as a tag.  unpack_method - Optional method used to reformatinbound data, called before anything else in on_next( ). (Default None) pack_method - Optional method used to reformat outbound data, calledbefore publishing in on_next( ). (Default None)  ″″″ super(ScalarCusumAlgorithm, self)._init_(name,   unpack_method,pack_method)  self._h_param = H  self._v_param = V  #check for a stablesignal and force sums to 0  residuals_crossed_zero = self._cusum_pos <0. and self._cusum_neg < 0.  if self._cusum_pos < 0.:   self._cusum_pos= 0.  if self._cusum_neg < 0.:   self._cusum_neg = 0.   #notify stableif crossed 0   if residuals_crossed_zero:    returnAlarmEventType.Signal_Estimate_Stable   #alarm and reset if overthreshold   elif self._cusum_pos > self._h_param or \    self._cusum_neg > self._h_param:    self.last_cusum_pos spike =self._cusum_pos    self.last_cusum_neg spike = self._cusum_neg   self._cusum_pos = 0.    self._cusum_neg = 0.    returnAlarmEventType.Alarm_Occurred   #otherwise nothing happened   else:   return AlarmEventType.No_Event  def reset_h(self, new_h):   ″″″Resetsthe H design parameter to the value passed in.   keyword parameters:  new_h - The new_h design parameter to use.   ″″″   self._h_param =new_h  def reset_v(self, new_v):   ″″″Resets the V design parameter tothe value passed in.   keyword parameters:   new_v - The new v designparameter to use.   self._v_param = new_v  def_reset_cusum(self):  ″″″Resets both cusum values to 0.″″″   self._cusum_pos = 0  self._cusum_neg = 0

The change data is indicative of one of stability or instability of thefiltered sensor data within the interval of time specified by the windowlength. The filtered sensor data may be deemed to be stable when valuesof the filtered sensor data are unchanging or less than a thresholdvalue during the interval of time. For example, where the values of thefiltered sensor data are within 5% of a median value, they may bedesignated as stable. The filtered sensor data may be deemed to beunstable when variance of the values exceeds the threshold value duringthe interval of time. For example, where a second value at an end of theinterval of time varies by more than 5% from a first value at abeginning of the interval of time, the filtered sensor data may beunstable. During periods of instability, the values in the filteredsensor data may be unreliable. For example, the unstable values may bethe result of mechanical motion while a platform of the inventorylocation 114 oscillates during settling. Use of the filtered sensor dataacquired during an unstable time may result in incorrect weight data126. Stated another way, the stable state may correspond to stableoutput by the weight sensor 120(6) (such as while settled), while theunstable state may correspond to unstable output by the weight sensor120(6) (such as during or immediately after a perturbation). By usingthe filtered sensor data acquired during a stable interval of time, theaccuracy of the weight data 126 used for subsequent processing isimproved. By utilizing the change data, the weight processing module maybe able to more quickly determine weight data 126 without having to waitfor a lengthy settling time.

Based on the change data, a first stable state occurring before anunstable state is determined. Also determined based on the change datais a second stable state after the unstable state.

Block 1216, based on the change data, determines a first stable statebefore an unstable state and a second stable state after the unstablestate. The transition from stable to unstable and back to stable may bedeemed indicative of an event or other activity at the inventorylocation 114.

Block 1218 determines the portion of the filtered sensor data associatedwith one or more of a first filtered sensor data associated with thefirst stable state or second filtered sensor data associated with thesecond stable state. For example, the change data may be associated withthe filtered sensor data using a time value. In some implementations,the association may be made using the last time value appearing withinthe range of time values corresponding to the respective state. Forexample, the time value of the last sample in the window length may beused.

Block 1220 determines final weight data 126. For example, the finalweight data 126 may comprise a weight difference indicative of a changein the weight from the first filtered sensor data to the second filteredsensor data. In another example, the weight data 126 may comprise thesecond filtered sensor data indicating the weight after return to astable state.

The process 1200 in some implementations may be iterated. For example,as additional samples of weight data 126 are determined and accessed,they may be included into the interval of time specified by the windowlength, while oldest samples may be dropped. Continuing the example, thewindow of time specified by the window length may “move forward” suchthat new samples of weight data 126 are added and oldest entry samplesare discarded from the window of time.

The processes discussed herein may be implemented in hardware, software,or a combination thereof. In the context of software, the describedoperations represent computer-executable instructions stored on one ormore non-transitory computer-readable storage media that, when executedby one or more processors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular abstract data types. Those havingordinary skill in the art will readily recognize that certain steps oroperations illustrated in the figures above may be eliminated, combined,or performed in an alternate order. Any steps or operations may beperformed serially or in parallel. Furthermore, the order in which theoperations are described is not intended to be construed as alimitation.

Embodiments may be provided as a software program or computer programproduct including a non-transitory computer-readable storage mediumhaving stored thereon instructions (in compressed or uncompressed form)that may be used to program a computer (or other electronic device) toperform processes or methods described herein. The computer-readablestorage medium may be one or more of an electronic storage medium, amagnetic storage medium, an optical storage medium, a quantum storagemedium, and so forth. For example, the computer-readable storage mediamay include, but is not limited to, hard drives, floppy diskettes,optical disks, read-only memories (ROMs), random access memories (RAMs),erasable programmable ROMs (EPROMs), electrically erasable programmableROMs (EEPROMs), flash memory, magnetic or optical cards, solid-statememory devices, or other types of physical media suitable for storingelectronic instructions. Further, embodiments may also be provided as acomputer program product including a transitory machine-readable signal(in compressed or uncompressed form). Examples of transitorymachine-readable signals, whether modulated using a carrier orunmodulated, include, but are not limited to, signals that a computersystem or machine hosting or running a computer program can beconfigured to access, including signals transferred by one or morenetworks. For example, the transitory machine-readable signal maycomprise transmission of software by the Internet. Separate instances ofthese programs can be executed on or distributed across any number ofseparate computer systems. Thus, although certain steps have beendescribed as being performed by certain devices, software programs,processes, or entities, this need not be the case, and a variety ofalternative implementations will be understood by those having ordinaryskill in the art.

Additionally, those having ordinary skill in the art readily recognizethat the techniques described above can be utilized in a variety ofdevices, environments, and situations. Although the subject matter hasbeen described in language specific to structural features ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features or acts described. Rather, the specific features andacts are disclosed as illustrative forms of implementing the claims.

What is claimed is:
 1. A system comprising: a shelf to hold one or moreitems; a camera with a field-of-view comprising at least a portion ofthe shelf; a sensor to generate sensor data for the shelf; and acomputing device comprising: a memory storing computer-executableinstructions; and a hardware processor to execute thecomputer-executable instructions to: access image data acquired by thecamera; determine an interaction has occurred at the shelf involving theone or more items based on the image data; determine a start time and anend time of the interaction; determine, based on at least a portion ofthe image data, a first set of hypotheses for the interaction, eachhypothesis of the first set of hypotheses comprising: one or more firstitem identifiers, and a first probability value that the one or morefirst item identifiers were involved in the interaction; access thesensor data generated by the sensor, the sensor data comprisingnon-image data; determine, based on the sensor data, a second set ofhypotheses for the interaction, each hypothesis of the second set ofhypotheses comprising: one or more second item identifiers, and a secondprobability value that the one or more second item identifiers wereinvolved in the interaction; determine that the first set of hypothesesand the second set of hypotheses do not meet or exceed a confidencethreshold value; generate a third set of hypotheses for the interactionby combining the first set of hypotheses and the second set ofhypotheses; determine at least one hypothesis of the third set ofhypotheses has a confidence value that meets or exceeds the confidencethreshold value; select a hypothesis from the third set of hypotheses asa solution to describe the interaction; and update inventory quantitiesfor the one or more items on the shelf using the solution.
 2. The systemof claim 1, the hardware processor to further execute thecomputer-executable instructions to: access item data; access a firstimage of the shelf at the start time of the interaction; access a secondimage of the shelf at the end time of the interaction; determine a firstcount of items at the start time of the interaction based on the firstimage and the item data; determine a second count of items at the endtime of the interaction based on the second image and the item data; anddetermine a quantity of items added to or removed from the shelf bysubtracting the first count of the items from the second count of theitems.
 3. The system of claim 1, wherein: the sensor comprises a weightsensor; the sensor data comprises weight data; and the third set ofhypotheses is generated using Bayes' rule.
 4. A system comprising: acomputing device comprising: a memory storing computer-executableinstructions; and a hardware processor to execute thecomputer-executable instructions to: access image data acquired by acamera with a field-of view inclusive of an inventory location, theinventory location stowing one or more items; determine an interactioninvolving one or more of the one or more items at the inventory locationhas occurred, based on one or more images of the image data; determine,based on at least a portion of the image data, a first set of hypothesesfor the interaction, each hypothesis of the first set of hypothesesindicative of a first probability that one or more first items wereinvolved in the interaction; access sensor data acquired by one or moresensors, the one or more sensors configured to generate the sensor datafor the inventory location; determine, based on the sensor data, asecond set of hypotheses for the interaction, each hypothesis of thesecond set of hypotheses indicative of a second probability that one ormore second items were involved in the interaction; determine that thefirst set of hypotheses and the second set of hypotheses do not meet orexceed a confidence threshold value; generate a third set of hypothesesfor the interaction by combining at least a portion of the first set ofhypotheses and at least a portion of the second set of hypotheses;determine that at least one hypothesis in the third set of hypotheseshas a probability value that meets or exceeds the confidence thresholdvalue; select a hypothesis from the third set of hypotheses as asolution that describes the interaction; and update inventory quantitiesof one or more of the one or more items stowed at the inventory locationbased on the solution.
 5. The system of claim 4, the hardware processorto further execute the computer-executable instructions to: access afirst image of the image data acquired before the interaction occurs;access a second image of the image data acquired after the interactionoccurs; and determine differential data indicative of a change at theinventory location using the first image and the second image.
 6. Thesystem of claim 5, wherein the computer-executable instructions todetermine the differential data indicative of the change at theinventory location further comprise instructions to: determine thechange at the inventory location occurs at a partitioned area of theinventory location; and determine one or more of addition or removal ofitems from the partitioned area of the inventory location.
 7. The systemof claim 5, wherein the computer-executable instructions to determinethe differential data indicative of the change at the inventory locationfurther comprise instructions to: access item data; determine a startingcount of items appearing in the first image; determine an ending countof items appearing in the second image; and determine a quantity ofitems added to or removed from the inventory location by subtracting thestarting count from the ending count.
 8. The system of claim 5, whereinthe computer-executable instructions to determine the differential dataindicative of the change at the inventory location further compriseinstructions to: detect first indicia or markings at the inventorylocation at a beginning of the interaction using the first image,wherein the first indicia or markings comprise one or more of: trackingmarks, bar codes, or a ruler printed on a surface of the inventorylocation; detect second indicia or markings at the inventory location atan end of the interaction using the second image; and compare the firstindicia or markings and the second indicia or markings to determinewhether items have been added to or removed from the inventory location.9. The system of claim 4, the hardware processor to further execute thecomputer-executable instructions to: determine the interaction involvesa change to at least one of a plurality of partitioned areas of theinventory location; access item data to determine items stowed at the atleast one of the plurality of partitioned areas of the inventorylocation; and wherein the first set of hypotheses is determined usingthe at least a portion of the image data and the item data.
 10. Thesystem of claim 4, wherein: the third set of hypotheses is generatedusing Bayes' rule.
 11. The system of claim 4, the hardware processor tofurther execute the computer-executable instructions to: determine,based on the solution, interaction data indicative of one or more of: achange in quantity of the one or more of the one or more items at theinventory location, an item identifier indicative of the one or more ofthe one or more items involved in the interaction, or at least onepartitioned area of the inventory location from which the one or more ofthe one or more items were picked or placed.
 12. The system of claim 4,the hardware processor to further execute the computer-executableinstructions to: determine occurrence of motion at a partitioned area ofthe inventory location based on the image data.
 13. The system of claim12, wherein the computer-executable instructions to determine theoccurrence of motion at the partitioned area of the inventory locationfurther comprise instructions to: determine a gradient differencebetween pixels in consecutive frames; and determine the occurrence ofmotion at the partitioned area of the inventory location based on thegradient difference exceeding a threshold value.
 14. The system of claim4, wherein: the one or more sensors comprise one or more weight sensors;the sensor data comprises weight data; and the each hypothesis of thesecond set of hypotheses is further indicative of a predicted quantityof the one or more second items involved in the interaction based on theweight data.
 15. The system of claim 4, the hardware processor tofurther execute the computer-executable instructions to: rank thehypotheses in the third set of hypotheses according to theircorresponding probabilities; and select the one hypothesis in the thirdset of hypotheses having a highest ranking as the solution.
 16. A methodcomprising: detecting an interaction involving one or more items stowedat an inventory location; accessing first sensor data acquired by afirst sensor collecting data about the inventory location during theinteraction and second sensor data acquired by a second sensorcollecting data about the inventory location during the interaction;determining, based on the first sensor data, a first set of hypothesesfor the interaction, each hypothesis in the first set of hypothesesindicative of a first probability that one or more first items wereinvolved in the interaction; determining, based on the second sensordata, a second set of hypotheses for the interaction, each hypothesis inthe second set of hypotheses indicative of a second probability that oneor more second items were involved in the interaction; determining thatthe first set of hypotheses and the second set of hypotheses do not meetor exceed a confidence threshold value; generating a third set ofhypotheses for the interaction by combining at least a portion of thefirst set of hypotheses and at least a portion of the second set ofhypotheses; determining that at least one hypothesis in the third set ofhypotheses has a probability value that meets or exceeds the confidencethreshold value; selecting a hypothesis from the third set of hypothesesas a solution for the interaction; and updating inventory quantities forthe one or more items stowed at the inventory location using thesolution.
 17. The method of claim 16, wherein the first sensor datacomprises image data from a camera with a field-of-view that includes atleast a portion of the inventory location; and the method furthercomprising: accessing a first image at a start of the interaction;accessing a second image at an end of the interaction; and determining aquantity of one or more of the one or more items added to or removedfrom the inventory location by comparing the first image and the secondimage.
 18. The method of claim 16, wherein the first sensor datacomprises image data from a camera with a field-of-view that includes atleast a portion of the inventory location; and the method furthercomprising: determining a start time and an end time for the interactionbased on at least a portion of the image data.
 19. The method of claim16, wherein the first sensor data comprises image data from a camerawith a field-of-view that includes at least a portion of the inventorylocation; and the method further comprising: determining the interactioninvolves a change to at least one partitioned area of the inventorylocation; accessing item data inclusive of items stowed at the at leastone partitioned area of the inventory location; and wherein the firstset of hypotheses is determined based on at least a portion of the imagedata and the item data.
 20. The method of claim 16, wherein: the firstsensor data comprises image data from a camera with a field-of-view thatincludes at least a portion of the inventory location and the secondsensor data comprises weight data from a weight sensor; and the thirdset of hypotheses is generated using Bayes' rule.